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Abstract 

The  Air  Force  is  developing  and  implementing  an  enterprise  architecture  to  meet 
the  Clinger-Cohen  Act’s  requirement  that  all  federal  agencies  use  an  architecture  to  guide 
their  IT  investments.  However,  this  act  does  not  provide  guidance  on  how  to  effectively 
manage  an  enterprise  architecture.  Prior  research  applied  maturity  models  and 
competency  stages  to  manage  an  enterprise  architecture  by  defining  layers  of  enterprise 
architecture  management  maturity.  However,  these  efforts  tend  to  view  enterprise 
architecture  development  as  a  one-time  planning  process  rather  than  an  iterative 
progression. 

Enterprise  architecture  is  not  a  one-time  exercise,  but  rather  it  is  an  on-going 
effort  within  the  organization  to  rationalize,  integrate,  and  optimize  the  IT  capability 
within  an  organization  across  many  projects  and  business  units.  Hence,  the  critical 
success  factors  to  effectively  manage  an  enterprise  architecture  must  be  identified  to 
ensure  the  structure,  processes,  and  governing  mechanisms  are  established  within  the 
organization  for  maintaining  an  enterprise  architecture. 

This  research  draws  from  existing  academic,  professional,  and  government 
literature  to  identify  the  key  issues  affecting  the  Air  Force's  ability  to  manage  its 
enterprise  architecture  effectively.  Once  identified,  a  quantitative  analysis  will  assist  in 
interpreting  the  qualitative  findings  in  hopes  of  determining  the  underlying  factors 
driving  these  issues. 
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EFFECTIVELY  MANAGING  THE 


AIR  FORCE  ENTERPRISE  ARCHITECTURE 


I.  Introduction 


Overview 

In  1884,  construction  began  on  the  Winchester  Mystery  House  in  San  Jose, 
California.  When  construction  was  completed,  the  house  consisted  of  160  rooms  and 
24,000  square  feet  of  living  space.  There  was  no  blueprint  for  this  construction  project; 
therefore,  it  took  the  147  builders  38  years  to  erect  this  house  at  a  cost  of  $5.5  million 
(equivalent  to  $165  million  today).  Without  a  master  plan,  there  was  no  orchestration  of 
the  innovative  skill  and  talent  used  to  construct  this  house.  In  the  end,  13  stairways  led  to 
nowhere,  65  doorways  opened  to  blank  walls,  24  skylights  were  embedded  in  the  floors, 
and  one  chimney  rose  the  entire  height  of  the  house  only  to  stop  short  of  the  roof  by  two 
feet  (Wennergren,  2004). 

To  effectively  design  and  construct  a  building,  a  blueprint  must  be  developed  and 
maintained.  A  blueprint  consists  of  a  set  of  drawings  that  defines  the  various 
characteristics  of  the  building.  Each  drawing  is  complementary  of  the  others  and 
provides  a  different  view  of  the  construction  project.  Therefore,  the  blueprint  results  in  a 
framework,  which  allows  architects,  engineers,  and  construction  personnel  with  divergent 
skill  sets  to  “speak”  a  common  language.  This  framework  allows  communication  to 
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become  more  efficient  and  creates  an  effective  roadmap  for  transforming  raw  materials 
into  a  finished  structure. 

The  Air  Force  is  employing  a  blueprint  known  as  the  Enterprise  Architecture 
Framework  (AF  Chief  Architect's  Office,  2003).  This  framework  permits  the  business, 
combat  support,  and  combat  operations  organizations  to  “speak”  a  common  language. 
More  importantly,  it  is  advancing  the  integration  of  its  operational  and  technological 
environments.  The  framework’s  descriptive  models  allow  decision  makers  to  understand 
the  complexities  around  how  the  two  environments  operate  today  and  how  they  should 
operate  in  the  future.  Just  like  a  construction  blueprint,  an  enterprise  architecture 
framework  can  provide  a  common  language  and  roadmap  that  clarifies  the 
interrelationships  among  enterprise  operations  and  the  underlying  Information 
Technology  (IT)  infrastructure. 

Motivation  for  Research 

Developing  and  implementing  an  enterprise  architecture  has  been  identified  as 
one  of  the  top  four  Information  Systems  (IS)  management  issues  since  1987  (Brancheau 
and  Wetherbe,  1987;  Niederman,  Brancheau  et  al.,  1991;  Brancheau,  Janz  et  al.,  1996). 
These  studies  recognized  there  was  no  overarching  framework  guiding  investments  in 
information  technology.  As  the  private  sector  focused  on  exploiting  their  enterprise 
architecture  to  integrate  its  IT  investments  with  its  business  objectives,  similar  studies 
conducted  on  public  agencies  have  proved  to  have  different  outcomes.  Swain  and  White 
identified  developing  an  enterprise  architecture  as  thirty-third  in  importance  among  top 
issues  of  public  IS  managers  (Swain,  White  et  al.,  1995). 
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One  year  later,  Section  5125  of  the  Clinger-Cohen  Act,  as  implemented  by 
Congress,  required  all  federal  agencies’  Chief  Information  Officers  to  develop, 
implement,  and  maintain  an  enterprise  architecture.  To  strengthen  the  enforcement  of 
this  act  the  Office  of  Management  and  Budget  (OMB),  under  the  guidance  of  the 
Government  Accounting  Office,  issued  Circular  A- 130  requiring  all  federal  agencies’ 
investments  in  information  systems  to  be  based  on  their  enterprise  architecture. 

Due  to  these  mandates,  the  Air  Force  has  developed  an  enterprise  architecture  to 
guide  its  investment  in  information  systems.  Furthermore,  it  provides  decision  makers 
with  the  ability  to  synchronize  mission  requirements,  programming,  budgeting,  and 
acquisition  management  with  infonnation  systems  planning.  To  implement  and  manage 
the  Air  Force  enterprise  architecture  the  Air  Force  Communications  Agency  (AFCA) 
transitioned  from  SCOPE  Network  teams  that  focused  on  optimizing  and  securing  the 
base  networks  to  SCOPE  EDGE  teams.  While  SCOPE  Net’s  mission  was  network 
optimization  SCOPE  EDGE  has  increased  the  level  of  responsibility  to  include  strategic 
network  advocacy  and  enterprise  level  impact  assessment  (Hoeft,  2004). 

In  response  to  this  change  in  mission,  this  research  effort  supports  AFCA  by 
identifying  the  key  issues  affecting  the  Air  Force's  ability  to  manage  its  enterprise 
architecture  effectively.  Once  identified,  a  quantitative  analysis  will  assist  in  interpreting 
the  qualitative  findings  in  hopes  of  detennining  the  underlying  factors  that  drive  these 
issues.  Therefore,  the  purpose  of  this  research  is  to  identify  and  analyze  the  key  issues 
affecting  the  Air  Force’s  ability  to  manage  its  enterprise  architecture. 
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Research  Question 

To  address  the  purpose  of  this  research,  the  following  central  organizing  research 
question  is  posited:  What  does  literature  identify  as  the  key  issues  affecting  the  Air 
Force's  ability  to  effectively  manage  its  enterprise  architecture? 

Investigative  Questions 

This  research  effort  will  address  multiple  investigative  questions  in  order  to 
answer  the  main  research  question: 

1 .  What  is  an  enterprise  architecture? 

2.  How  does  the  Air  Force  define  its  enterprise  architecture? 

3.  What  does  the  literature  identify  as  the  issues  that  must  be  addressed  to  effectively 

manage  an  enterprise  architecture? 

4.  Which  issues  have  the  most  relevance? 

5.  What  does  the  literature  identify  as  the  underlying  factors  driving  these  issues? 

Methodology 

A  sequential  exploratory  research  method  was  conducted.  Therefore,  this 
research  effort  was  completed  in  two  phases.  The  initial  phase  consists  of  collecting 
relevant  white  papers,  case  studies,  and  prior  empirical  and  exploratory  research  efforts. 
These  documents  cover  the  enterprise  architecture  and  information  system  (IS)  strategic 
planning  fields  of  study.  A  content  analysis  identified,  categorized  and  synthesized  the 
literature  to  discover  the  main  attributes  and  to  extract  themes  from  the  articles  regarding 
the  management  of  the  enterprise  architecture.  The  second  phase  quantitatively  analyzed 
the  identified  issues  the  Air  Force  must  assess  to  detennine  its  effectiveness  in  managing 
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its  enterprise  architecture.  Not  only  did  this  analysis  determine  the  most  relevant  issues, 
but  it  also  detennined  the  underlying  factors  driving  these  issues. 

Significance 

Prior  research  efforts  have  focused  on  identifying  the  key  issues  involved  in 
developing  and  implementing  an  enterprise  architecture  by  investigating  case  studies 
involving  the  implementation  of  enterprise  architectures  and  the  utilization  of  IS  strategic 
planning  (Brancheau  and  Wetherbe,  1986;  Earl  1993;  Kim  and  Everest,  1994;  Periasamy 
andFeeny,  1997;  Shanks,  1997;  Segars  and  Grover,  1998;  Ross,  2003).  However,  these 
studies  have  not  methodically  identified  and  analyzed  the  key  issues  in  successfully 
managing  an  enterprise  architecture  and  the  factors  driving  these  issues. 

This  research  effort  synthesizes  the  literary  efforts  of  a  wide  range  of  academic 
and  professional  authors.  This  synthesis  provides  AFCA  with  a  strategic  guidepost  to 
broaden  their  understanding  of  the  holistic  perspective  required  to  manage  the  Air 
Force’s  enterprise  architecture.  It  also  fulfills  the  identified  academic  void  by 
methodically  identifying  the  key  issues  surrounding  managing  an  enterprise  architecture. 

Thesis  Overview 

Chapter  I  supplies  an  overview  of  enterprise  architecture,  the  motivations  for 
research,  the  research  question,  investigative  questions,  a  description  of  the  study,  and  the 
significance  of  completing  this  research  effort.  The  remainder  of  this  thesis  reports  the 
efforts  to  address  the  research  questions  presented  in  this  chapter.  Chapter  2  defines  what 
is  an  enterprise  architecture,  followed  by  the  Air  Force’s  description  and  definition  of  its 
enterprise  architecture.  In  addition,  a  literature  review  of  enterprise  architecture  research 
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is  presented  to  provide  the  theoretical  foundations  of  this  research  effort.  Chapter  3 
expounds  upon  the  justification  for  the  methodology  used  along  with  a  systematic  guide 
explaining  how  the  content  analysis  was  performed.  Chapter  4  sets  forth  a  detailed 
analysis  of  the  collected  data  and  the  findings  that  resulted  from  the  employed 
methodology.  Finally,  Chapter  5  provides  conclusions,  limitations,  and 
recommendations  for  future  research. 


6 


II.  Literature  Review 


Overview 

This  review  examines  the  literature  relevant  to  the  research  topic  of  enterprise 
architecture.  From  this  examination  a  common  frame  of  reference  for  this  exploratory 
research  is  presented.  First,  an  enterprise  architecture  is  defined.  Then  a  description  of 
the  Air  Force’s  enterprise  architecture  is  covered  leading  to  an  operational  definition. 
Finally,  the  existing  research  in  the  field  of  enterprise  architecture  will  be  reviewed  to 
provide  the  theoretical  foundations  of  this  research. 

Defining  an  Enterprise  Architecture 

The  term  Enterprise  Architecture  (EA)  lacks  a  universally  accepted  definition. 
Prior  to  discussing  the  theoretical  foundation  of  this  research,  a  common  frame  of 
reference  is  established  by  presenting  an  operational  definition  of  an  enterprise 
architecture.  Until  1986,  there  was  little  consistency  among  the  concepts  and 
tenninology  regarding  enterprise  architectures.  In  response,  John  Zachman  presented  a 
conceptual  framework  for  defining  this  tenn. 

This  framework,  a  two-way  matrix  as  presented  in  Figure  1,  consists  of  six  views 
and  six  information  sources.  The  six  views  represent  the  perspective  of  each  participant 
included  in  the  enterprise  architecture  development  process.  Each  view  is  independent  of 
the  next.  Therefore,  the  level  of  detail  does  not  increase  with  each  successive  layer. 
Instead,  it  varies  within  each  participant’s  architectural  representation  (Zachman,  1987). 
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To  allow  each  participant’s  enterprise  architecture  representation  to  vary  six 
information  sources  are  presented  across  the  top  of  the  matrix.  Collectively,  these 
sources  comprise  each  level’s  description.  Just  as  the  perspectives  stand  alone,  so  do  the 
six  descriptions.  This  allows  the  participants  to  describe  the  same  product  in  multiple 
ways,  which  provides  them  with  the  ability  achieve  multiple  purposes  with  an  enterprise 
architecture  (Zachman,  1987). 


Figure  1.  Zachman’s  Framework 

Since  Zachman’s  seminal  research,  several  studies  have  made  attempts  to  further 
clarify  the  concepts  surrounding  enterprise  architecture.  Kim  and  Everest  expanded  on 
Zachman’s  definition  of  an  enterprise  architecture  by  presenting  four  sub-architectures: 
process,  data,  control,  and  technology  (Kim  and  Everest,  1994).  These  four  sub¬ 
architectures  link  IS  planning  with  the  corresponding  levels  of  Zachman’s  architecture. 
Similar  to  Zachman’s  framework,  the  views  complement  each  other  and  taken 
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collectively  present  an  IS  enterprise  architecture  that  provides  the  basis  for  constructing 
an  information  system  and  managing  information  resources  (Kim  and  Everest,  1994). 

Segars  and  Grover’s  describe  the  development  of  an  IS  architecture  as  a  three- 
level  hierarchy  of  analysis  and  development  and  can  be  summarized  as  follows  (Segars 
and  Grover,  1998): 

•  Conceptual  modeling — a  level  at  which  broad  business,  information,  process  and 
data  categories  and  interrelationships  are  identified 

•  Logical  systems  level — core  concepts  are  expanded,  structural  relationships 
between  architecture  components  are  mapped,  and  sufficient  detail  is  captured  to 
enable  the  identification  of  applications,  databases  and  core  business  processes 

•  Physical  level — logical  constructs  are  realized  in  operational  system  and 
databases,  within  the  constraints  imposed  by  specific  performance  and  topological 
physical  system  requirements 

This  description  concludes  that  the  development  of  an  IS  architecture  is  a  set  of  high- 
level  models  showing  corporate  data,  process  and  application  structures  in  logical  form, 
supported  by  a  set  of  corporate  definitions  of  core  data  and  process  components 
(Hamilton,  1999). 

Even  though  research  was  completed  to  refine  the  definition  of  an  enterprise 
architecture  terminology  such  as  architecture  and  infrastructure  were  still  being  used 
interchangeably.  This  stemmed  from  referring  to  the  enterprise  architecture  as  a  “city 
plan”  which  focuses  on  developing  detailed  drawings  of  the  interconnections  between 
processes,  infrastructure,  data,  and  applications  (Ross,  2003).  Using  the  enterprise 
architecture  in  this  fashion  does  not  capture  its  ability  to  tie  itself  to  the  organization’s 
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business  strategy.  Therefore,  Ross  provided  the  following  definition  of  an  enterprise 
architecture  (Ross,  2003): 

An  enterprise  architecture  is  the  organizing  logic  for  applications,  data,  and 
infrastructure  technologies,  as  captured  in  a  set  of  policies  and  technical  choices, 
intended  to  enable  the  firm’s  business  strategy. 

By  looking  across  each  of  these  conceptualizations  of  an  enterprise  architecture  a 
common  theme  presents  itself.  Therefore,  the  operational  definition  of  an  enterprise 
architecture  for  the  purpose  of  this  research  is  the  organization  of  computing  resources  in 
an  organization,  which  consists  of  data,  information,  applications,  infrastructure,  and 
personnel  to  enable  a  firm’s  business  strategy. 

Defining  the  Air  Force’s  Enterprise  Architecture 

As  academic  literature  refined  the  definition  of  an  enterprise  architecture,  the 
federal  government  passed  public  laws  and  issued  directives  requiring  each  agency  to  use 
an  enterprise  architecture.  For  example,  OMB  Circular  A- 130  requires  each  agency  to 
use  an  EA  to  document  the  linkages  between  its  mission  needs,  information  content,  and 
information  technology  capabilities.  In  fulfilling  this  requirement  the  Department  of 
Defense  (DoD)  leveraged  Zachman’s  framework  to  develop  the  DoD  Architecture 
Framework  (DoDAF). 

The  DoDAF,  Version  1.0,  defines  a  common  approach  for  DoD  enterprise 
architecture  description  development,  presentation,  and  integration  for  both  warfighting 
operations  and  business  operations  and  processes  (DoD  Architecture  Framework 
Working  Group,  2004).  By  providing  a  common  foundation  this  framework  ensures 


10 


architecture  descriptions  can  be  compared  and  related  across  both  Joint  Service  and 
multinational  boundaries.  To  achieve  this  objective  Zachman’s  Enterprise  Architecture 
Framework  was  condensed  to  an  Operational  View,  a  Systems  View,  and  a  Technical 
Standards  View.  The  condensed  version  of  Zachman’s  Enterprise  Architecture 
Framework,  known  as  the  DoD  Architecture  Framework  is  presented  in  Figure  2.  As  can 
be  seen,  the  DoD  reduced  the  number  of  descriptions  for  each  layer  of  the  architecture 
from  six  to  three. 


Just  as  in  Zachman’s  Framework,  each  perspective  is  independent  of  the  next. 
Therefore,  the  level  of  detail  does  not  increase  by  traveling  down  through  the  successive 
layers  of  the  matrix.  However,  the  three  views  provide  a  means  to  model  their  respective 
architecture  components  according  to  the  requirements  for  each  perspective.  Through 
this  decomposition,  several  diagrams  of  the  same  perspective  are  developed  allowing  the 
participant’s  perspective  to  be  described  in  multiple  ways.  Table  1  presents  a  summary 
of  the  three  views  contained  within  the  DoD’s  Architecture  Framework  (DoD 
Architecture  Framework  Working  Group,  2004). 
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Table  1.  DoD  Architecture  Framework  Views 


View 

Definition 

Operational 

A  description  of  the  tasks  and  activities,  operational  elements,  and  information  exchanges 
required  to  accomplish  DoD  warfighting  missions  and  business  processes.  This  view 
identifies  operational  nodes  and  elements,  their  assigned  tasks  and  activities,  and 
information  flows  required  between  nodes. 

System 

A  set  of  graphical  and  textual  products  describing  systems  and  interconnections  that 
support  DoD  warfighting  and  business  functions.  This  view  associates  system  resources  to 
the  Operational  View  by  determining  which  system  resources  support  the  operational 
activities  and  facilitate  the  exchange  of  information  among  operational  nodes. 

Technical 

A  minimal  set  of  rules  governing  the  arrangement,  interaction,  and  interdependence  of 
system  parts  or  elements  to  ensure  a  system  satisfies  a  specified  set  of  operational 
requirements.  The  TV  provides  the  technical  systems  implementation  guidelines  upon 
which  engineering  specifications  are  based,  common  building  blocks  are  established,  and 
product  lines  are  developed. 

The  AF  Enterprise  Architecture  Framework  (AF-EAF)  tailored  and  refined  the 
standards  set  by  the  Department  of  Defense  Architecture  Framework  for  use  by  the  Air 
Force.  The  AF-EAF  is  a  taxonomy  of  pertinent  information  used  to  systematically 
describe  and  document  the  Air  Force  Enterprise  Architecture  (AF-EA).  This  taxonomy  is 
a  key  construct  in  the  advancement  of  the  Air  Force’s  integration  efforts,  allowing  for  the 
interoperability  among  the  Air  Force’s  and  the  Joint  Services’  infonnation  systems. 
Additionally,  the  architecture  acts  as  a  tool  allowing  the  Air  Force’s  vision,  mission,  and 
operational  concept  planning  to  be  fully  synchronized  with  programming,  budgeting,  and 
acquisition  management  (Roche,  2002). 

The  AF-EAF  does  not  define  the  Air  Force’s  Enterprise  Architecture.  Instead,  the 
framework  is  a  tool  that  is  used  to  present  the  various  models,  perspectives,  and 
definitions  for  communicating  the  architecture’s  components.  As  shown  in  Figure  3,  the 
AF-AEF  consists  of  three  parts.  Each  part  is  used  as  a  communication  tool  to  establish  a 
common  foundation  for  integrating  architectures  (AF  Chief  Architect's  Office,  2003). 
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Figure  3:  Air  Force  Enterprise  Architecture  Framework 


Architecture  Drivers  and  Inputs 

One  of  the  main  drivers  of  the  AF-EA  is  the  Chief  of  Staff  of  the  Air  Force’s  six 
Concepts  of  Operation  (CONOPs).  These  CONOPs  provide  the  strategic  direction  for 
the  development  of  the  AF-EA.  In  turn,  the  enterprise  architecture  is  leveraged  as  a 
foundation  allowing  for  the  integration  of  business  and  combat  support  elements  with 
each  other  along  with  combat  operations  (Roche,  2002).  The  establishment  of  this 
relationship  was  directly  influenced  by  an  assortment  of  public  laws,  policies,  directives, 
and  architecture  governance  direction.  Table  2,  located  below,  provides  a  synopsis  of  the 
drivers  that  directly  affect  the  development  of  the  AF-EA. 
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Table  2.  Public  Law,  Directives,  and  Instructions 


Public  Law 

Date 

Description 

Government  Performance 
and  Reform  Act  (GPRA) 

1993 

Sets  the  stage  for  additional  Information  Resource  Management  reform 

Clinger-Cohen  Act 

1996 

Requires  government  agencies  to  develop  and  use  architectures  to 
integrate  information  technology  with  their  business  processes 

E-Govemment  Act  of 
2002 

2002 

Enhances  the  management  and  promotion  of  electronic  government 
services  and  processes  by  providing  a  framework  of  measures  and 
establishing  a  Federal  CIO  within  the  OMB 

OMB  Circular  A-130 

2002 

States  federal  agencies  that  do  not  utilize  an  enterprise  architectures  in 
support  of  strategic  planning  will  not  receive  federal  funding 

Directives 

Date 

Description 

DoD  Directive  5000.1 

2003 

Require  all  services  to  develop  joint  capability  integrated  architectures 
and  DoD  component  functional  area  integrated  architectures  that  are 
documented  using  the  DoD  Architecture  Framework. 

CJCSI  3170. 01C 

Draft 

The  instruction  replaces  CJCSI  3170.01B  stating  joint  concepts  and 
supporting  architectures  will  serve  as  the  basis  for  evaluating  and 
approving  all  future  joint  and  service  capabilities  proposals. 

DoDI  8400. xx 

Draft 

It  will  require  DoD  Component  architectures  to  be  developed  and 
maintained  consistent  with  the  Global  Information  Grid  architecture, 
direct  the  use  of  the  DoDAF,  and  implement  a  standard  approach  and 
data  requirement  for  architecture  development  using  the  DoD  Core 
Architecture  Data  Model. 

Air  Force  Instruction 
(AFI)  33-124 

2000 

Supports  the  architecture-related  mandates  of  the  Clinger-Cohen  Act  of 
1996 ,  Information  Technology’  Management  Reform  Act ,  and 
promulgates  Air  Force  Enterprise  C4ISR  architecture  products 
identified  in  the  DoDAF. 

To  comply  with  this  guidance  the  Air  Force  reviewed  strategic  plans,  CONOPs, 
capability  documents,  and  task  list.  From  this  review  the  Air  Force  Enterprise 
Architecture  was  broken  into  three  distinct  components.  As  seen  in  Figure  4,  each 
functional  area  was  identified  as  either  a  warfighting  or  support  role  (Fore,  2000). 
Moreover,  the  support  role  was  further  divided  into  combat  and  business  support.  The 
two  mission  areas,  warfighting  and  support,  were  then  leveraged  to  develop  a  supporting 
sub-architecture.  To  bridge  these  two  architectures  a  third  infostructure  architecture  was 
developed  to  ensure  the  three  elements  were  capable  of  integrating  their  diverse 
requirements  (AF  Communications  Agency,  2003). 
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Figure  4.  Air  Force  Enterprise 


Perspectives 

The  side  effect  of  developing  the  Air  Force  Enterprise  Architecture  was  the 
creation  of  functionally  oriented  stakeholders.  Therefore,  three  separate,  but  related 
categories  of  AF-EA  products  and  artifacts  have  been  developed.  Each  of  the  categories, 
see  Figure  5,  is  comprised  of  components  that  are  relevant  to  a  particular  group  of 
stakeholders  (AF  Chief  Architect's  Office,  2003). 
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Enterprise 

The  perspective  focusing  on  strategic  plans,  enterprisewide  processes,  key 
information  and  infrastructure  important  to  the  enterprise,  and  a  frarrew  ork  to  enable 
lower  level  architectures  to  be  relatable  to  other  architectures  that  together  make  up 
the  enterprise  architecture. 

Mission  & 
Cross  Mission 
Area 

The  perspective  focusing  on  a  subset  of  the  enterprise  defined  by  a  specific  mission, 
function,  business  area,  or  set  of  capabilities,  activities,  or  shared  data.  This 
perspective  is  primarily  operationally  focused  and  is  user/operator  “centric”. 

Program 
&  Node 

V 

The  perspective  focusing  on  an  individual  system  or  a  group  of  systems  and  the 
interelationships  with  othersystems.  This  perspective  is  primarily  system  focused 
and  is  program  manager  or  node  manager  “centric”. 

Figure  5.  Perspectives  of  the  AF-EA 
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Uses  and  Impacts 

Collectively,  these  perspectives  allow  the  participants  to  utilize  the  architecture  in 
multiple  fashions.  The  following  are  examples  of  how  the  enterprise  architecture  is 
exploited  to  ensure  the  Air  Force  can  achieve  its  core  processes  through  capability  based 
planning  (AF  Chief  Architect's  Office,  2003): 

•  Supports  the  Planning,  Programming,  and  Budgeting  System  (PPBS) — the 
PPBS  must  allocate  funds  across  competing  programs  and  activities.  To  ensure  the 
IT  strategy  provides  the  maximum  benefit  to  the  Air  Force  the  AF-EA  is  used  to 
provide  the  necessary  context  while  making  capability-based  decisions. 

•  Supports  Joint  Capabilities  Integration  and  Development — the  joint 
capabilities  integration  and  development  process  produces  the  warfighters’  projected 
capability  needs.  During  this  process,  information  is  required  on  the  current  and 
planned  capabilities  of  existing  information  systems.  The  AF-AE  provides  this 
information  by  documenting  system  interdependencies,  capturing  the  functionality 
resident  in  each  existing  or  planned  systems,  and  identifying  gaps  or  deficiencies  that 
prevent  the  AF  from  achieving  critical  capabilities  or  mission  needs. 

•  Supports  the  Acquisition  Process — The  AF-EA  guides  and  constrains  system 
developers  to  ensure  the  resulting  system  is  interoperable  with  the  remainder  of  the 
systems  and  applications  that  make  up  the  enterprise  architecture. 

•  Supports  the  Planning  and  Operations  processes — The  AF-EA  is  also  used  to 
support  warfighter  contingency  planning.  The  AF-EA  gives  Combatant  Commanders 
a  set  of  operational  and  system  view  products  that  define  existing  IT  capabilities  and 
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limitations.  In  addition,  the  AF-EA  provides  a  basis  for  rapid  reconfiguration  of 
architectures  to  meet  needs  of  contingency  operations. 

An  overall  description  of  the  three  separate  components  comprising  the  AF-EAF 
has  been  provided.  The  ultimate  goal  of  providing  this  description  is  to  determine  how 
the  Air  Force  defines  its  enterprise  architecture.  As  stated  before,  the  AF-EAF  does  not 
define  the  Air  Force’s  enterprise  architecture.  Instead,  the  framework  is  only  a  tool  used 
to  present  the  various  models,  perspectives,  and  definitions  for  communicating  the 
architecture’s  components.  However,  from  this  taxonomy  the  following  operational 
definition  of  the  Air  Force  Enterprise  Architecture  will  be  used  for  the  purpose  of  this 
research: 

The  Air  Force  Enterprise  Architecture  is  a  tool  that  allows  the  vision,  mission,  and 
operational  planning  to  be  synchronized  with  programming,  budgeting,  and 
acquisition  management  to  achieve  the  Air  Force’s  strategic  objectives. 

As  stated  in  Chapter  1,  the  implementation  and  management  of  the  Air  Force 
Enterprise  Architecture  has  become  the  responsibility  of  the  Air  Force  Communications 
Agency  (AFC  A).  In  response  to  this  new  mission,  one  of  the  actions  taken  by  AFC  A  was 
to  alter  their  SCOPE  Network  teams  into  SCOPE  EDGE  teams.  While  SCOPE  Net’s 
mission  was  strategic  network  advocacy  SCOPE  EDGE  has  increased  AFCA’s  level  of 
responsibility  to  include  enterprise  level  assessment  (Hoeft,  2004).  The  implications  of 
this  change  in  mission  focus  can  be  understood  by  taking  a  look  back  at  this  unit’s 
formation  and  subsequent  modification  throughout  its  existence. 

In  1968,  the  Air  Force  developed  Project  Scope  Creek.  The  focus  of  this  project 
was  to  apply  “scientific  methods  to  determine  system  capability,  isolate  faults,  and  make 
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corrections  or  modifications  to  make  equipment  perform  “like  new”:  (Air  Force 
Communications  Agency,  2004).  In  1973,  the  success  of  this  project  lead  to  the 
application  of  these  same  scientific  methods  to  systematically  evaluate  the  Air  Force’s 
communication  network. 

Between  1973  and  1997  the  Air  Force  experienced  a  growth  in  the  need  for  data 
transport  to  meet  the  mission  requirements  of  both  the  warfighter  and  combat  support 
elements.  By  1997,  the  large  number  of  local  area  networks,  metropolitan  area  networks, 
and  wide  area  networks  caused  AFCA  to  revise  the  Scope  Creek  concept  into  Scope 
Network.  The  mission  of  this  newly  formed  unit  was  to  focus  on  network  optimization 
(Air  Force  Communications  Agency,  2004).  To  achieve  this  mission  Scope  Network 
developed  and  utilized  a  Network  Maturity  Model.  This  model  provided  a  framework  for 
assisting  bases  in  developing  more  mature  networks  capable  of  meeting  a  base’s  mission 
requirements.  From  this  framework  Scope  Network  had  the  ability  to  assess  a  base 
network’s  current  level  of  maturity  and  then  detennine  the  necessary  requirements  to 
optimize  the  base  network  (Air  Force  Communications  Agency,  1998). 

AFCA’s  requirement  to  implement  and  manage  the  Air  Force  Enterprise 
Architecture  expanded  Scope  Network’s  view  of  the  network  as  a  collection  of  individual 
base  networks  to  SCOPE  EDGE’s  current  view  of  the  network  as  an  enterprise  (Hoeft, 
2004).  This  new  view  has  once  again  resulted  in  a  change  in  mission  focus  from 
ensuring  an  individual  base’s  network  is  optimized  to  ensuring  the  standardization  of  the 
Air  Force  networks. 

As  can  be  seen,  even  though  AFCA’s  level  of  responsibility  has  expanded  to 
include  the  entire  Air  Force  Enterprise  Architecture,  SCOPE  EDGE  is  currently  focusing 
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on  the  network  architecture.  This  technological  perspective  is  operationalized  by  the 
continued  used  of  the  Network  Maturity  Model  and  SCOPE  EDGE’s  mission  statement 
(Air  Force  Communications  Agency,  2004). 

Strengthen  enterprise  standardization  through  compliance  assessments,  network 
optimization,  reconstitution,  and  feedback  into  the  Air  Force  network  architecture. 
This  focus  only  accounts  for  the  technological  view  of  the  Air  Force’s  Enterprise 
Architecture  causing  an  emphasis  to  be  placed  on  the  hardware  and  software  that 
comprise  the  enterprise  architecture,  but  at  the  same  time  neglects  all  the  other 
architecture  views. 

Theoretical  Foundation 

In  2002,  the  General  Accounting  Office  (GAO)  reported  52  percent  of  federal 
agencies  have  satisfied  the  requirement  for  developing  enterprise  architectures. 

However,  only  four  percent  report  of  all  the  federal  agencies  satisfied  the  management 
practices  necessary  for  effective  enterprise  architecture  management  (General 
Accounting  Office,  2002).  In  response,  the  GAO  recognized  that  the  ability  to  effectively 
manage  an  enterprise  architecture’s  development,  maintenance,  and  use  depends  upon 
having  meaningful  measures  of  that  activity  in  relation  to  some  standard  (General 
Accounting  Office,  2003). 

The  ability  to  measure  the  salient  issues  when  managing  the  enterprise 
architecture  permits  managers  to  assess  progress  toward  the  desired  end  and  to  take 
corrective  action  to  address  unacceptable  deviations.  Ross  defines  the  desired  end  as  the 
objectives  of  the  enterprise  architecture  specifying  what  the  architecture  enables  the 
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business  to  do  (Ross,  2003).  Prior  studies  assessed  the  ability  to  use  a  Capability 
Maturity  Model  as  a  tool  to  measure  the  level  of  maturity  of  an  organization’s  enterprise 
architecture  (Thow-Yick,  1993;  Patnayakuni  and  Rai,  2002;  Jeffery  and  Leliveld,  2004). 

The  question  remains,  within  the  capability  maturity  model  what  are  the  main 
issues  that  must  be  addressed  to  effectively  manage  an  enterprise  architecture.  An 
attempt  was  made  by  the  OMB  to  answer  this  question  by  presenting  an  assessment 
framework  consisting  of  three  components:  1)  hierarchical  stages  of  management 
maturity,  2)  categories  of  attributes  that  are  critical  to  the  success  in  managing  any 
endeavor,  and  3)  elements  of  EA  management  that  form  the  core  of  the  EA  management 
practice  (General  Accounting  Office,  2003). 

In  contrast  to  this  framework,  Ross  contends  the  process  of  developing  an 
enterprise  architecture  is  not  an  orderly  endeavor.  Instead,  she  identifies  four  progressive 
competency  stages  that  an  organization  must  achieve  to  effectively  manage  their 
enterprise  architecture  (Ross,  2003).  Allen  and  Boynton  reached  a  similar  conclusion  by 
addressing  how  information  systems  architecture  can  be  used  to  support  organizations. 
They  reviewed  two  architectural  solutions’  benefits  and  pitfalls.  In  the  end,  they  contest 
neither  enterprise  architecture  will  succeed  on  its  own.  Instead,  firms  must  combine 
elements  of  both  to  meet  the  challenges  of  integrating  their  information  technology  with 
organizational  strategy  and  structure  (Allen  and  Boynton,  1991). 

These  two  research  efforts  point  to  the  fact  that  there  are  variations  in  the  results 
of  utilizing  an  enterprise  architecture.  In  support  of  this  claim,  Chalmeta  and  Campos 
found  that  each  enterprise  architecture  must  be  adapted  to  the  needs  of  the  organization 
(Chalmeta,  Campos  et  ah,  2001).  Therefore,  managing  an  enterprise  architecture  is  not  a 
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stable  process  as  presented  in  OMB ’s  Enterprise  Architecture  Capability  Maturity 
Framework.  Instead,  this  process  is  an  ongoing  effort  to  rationalize,  integrate,  and 
optimize  the  information  system  capability  within  the  organization  across  many  projects 
and  business  units  (Fong  Boh,  Yellin  et  al.,  2003). 

Within  this  ongoing  effort,  an  organization’s  ability  to  effectively  manage  an 
enterprise  architecture  is  directly  influenced  by  its  ability  to  reach  a  competency  level 
where  infonnation  system  capabilities  shape  the  organization’s  strategy.  At  the  same 
time,  the  organization’s  strategy  must  be  able  to  mold  information  system  capabilities  in 
response  to  changes  in  the  market  conditions  and  organizational  realities  (Ross,  2003). 

To  reach  this  level  an  organization  can  either  employ  a  capability  maturity  model  or 
focus  on  improving  its  competency.  However,  using  either  approach  requires  the 
identification  of  the  key  issues  to  successfully  managing  an  enterprise  architecture.  To 
date  the  current  stream  of  research  has  not  identified  these  key  issues  involved  in  the 
successful  management  of  an  enterprise  architecture  or  the  underlying  factors  driving 
these  issues. 

Therefore,  the  focus  of  this  research  is  twofold.  First,  it  will  fulfill  the  identified 
academic  void.  Furthermore,  as  the  Air  Force  Communication  Agency  takes  on  its  new 
mission  this  research  provides  them  with  a  guidepost  that  can  be  utilized  to  broaden  their 
focus  from  only  the  network  architecture  to  a  holistic  perspective.  This  new  perspective 
will  assist  AFC  A  by  identifying  the  key  issues  and  their  respective  underlying  factors  to 
effectively  manage  the  Air  Force’s  Enterprise  Architecture. 
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Summary 

This  chapter  provided  a  contextual  understanding  of  an  enterprise  architecture. 
Then  the  Air  Force’s  definition  of  an  enterprise  architecture  was  provided.  Finally,  a 
theoretical  discussion  explained  the  background  knowledge  required  to  understand  how 
prior  research  efforts  led  to  the  motivation  to  complete  this  current  study. 
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III.  Methodology 


Methodology  Overview 

As  with  any  research,  the  researcher  preserved  a  balance  between  maintaining  a 
realistic  perspective  and  ensuring  control  over  the  selected  methodology  (Mason, 
McKenney  et  ah,  1997).  The  researcher  established  the  realistic  perspective  in  Chapter  2 
by  providing  an  account  of  the  development  of  the  enterprise  architecture  concept.  This 
was  followed  up  with  a  historical  account  of  the  implementation  of  both  the  Department 
of  Defense’s  and  the  Air  Force’s  enterprise  architecture.  Providing  this  context  served  as 
the  necessary  background  information  to  formulate  answers  to  the  following  investigative 
questions: 

1 .  What  is  an  enterprise  architecture? 

2.  How  does  the  Air  Force  define  its  enterprise  architecture? 

To  answer  both  of  these  questions  the  researcher  had  to  gather  evidence, 
determine  patterns,  and  then  develop  an  agreed  upon  operational  definition  for  the 
purpose  of  this  research  (Mason,  McKenney  et  ah,  1997).  The  evidence  consisted  of 
academic  literature,  government  reports,  Air  Force  instructions,  and  policies.  Each  of 
these  was  reviewed  and  through  triangulation  patterns  were  identified.  These  two  steps 
allowed  the  researcher  to  reach  an  operational  definition  of  the  two  investigative 
questions. 

The  literature  review  also  explained  how  prior  research  efforts  have  only 
identified  issues  leading  to  the  success  or  failure  of  developing  and  implementing  an 
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enterprise  architecture.  However,  this  body  of  knowledge  has  not  addressed  how  to 
manage  an  enterprise  architecture  once  it  is  in  place. 

Research  Strategy 

Creswell  states,  if  a  concept  needs  to  be  understood  because  little  research  has 
been  done  on  it,  then  it  merits  a  qualitative  approach  (Creswell,  2003).  The  concept 
under  study  is  identifying  the  key  issues  affecting  the  Air  Force’s  ability  to  manage  its 
enterprise  architecture.  Qualitative  research  can  never  capture  objective  reality;  as  a 
result,  the  use  of  mixed  methods,  or  triangulation,  reflects  an  attempt  to  secure  an  in- 
depth  understanding  of  the  phenomenon  in  question  (Denzin  and  Lincoln,  2000).  This 
research  secures  an  in-depth  understanding  of  the  phenomenon  in  question  by  utilizing 
quantitative  data  to  assist  in  the  interpretation  of  qualitative  findings  (Creswell,  2003).  In 
short,  the  use  of  a  mixed  method  research  strategy  not  only  identifies  the  key  issues,  but 
also  reveals  the  most  relevant  ones. 

The  selected  mixed  method  consists  of  two  separate  phases.  The  first  phase 
encompasses  a  qualitative  content  analysis  to  identify,  categorize  and  synthesize 
literature  pertaining  to  enterprise  architecture,  enterprise  infrastructure,  systems 
development,  and  strategic  data  planning.  By  using  a  coding  schema,  a  systematic 
examination  of  the  data  was  conducted  to  identify  core  consistencies  or  themes  (Patton, 
2002)  and  answer  the  third  investigative  question  of  this  study: 

3.  What  does  the  literature  identify  as  the  issues  that  must  be  addressed  to  effectively 
manage  an  enterprise  architecture? 
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Once  the  issues  were  identified,  they  were  evaluated  using  relevant  statistical 
measurements  in  an  effort  to  make  inferences  about  the  qualitative  findings.  Therefore, 
the  second  phase  included  an  interpretation  of  the  overriding  themes,  which  should 
answer  the  final  two  investigative  question  of  this  study: 

4.  Which  issues  have  the  most  relevance? 

5.  What  does  the  literature  identify  as  the  underlying  factors  driving  these  issues? 

By  employing  this  two-phased  mixed  method,  the  key  issues  affecting  the  Air 

Force’s  ability  to  manage  its  enterprise  architecture  are  identified  and  analyzed.  The 
remainder  of  this  chapter  explains  the  deductive  process  of  selecting  this  sequential 
exploratory  research  strategy  and  the  methodology  employed  to  achieve  the  researcher’s 
overall  objective. 

Mixed  Method  Approach 

Determining  which  mixed  method  to  employ  for  this  research  required  four 
questions  to  be  answered  (Creswell,  Plano  et  al.,  2003): 

1 .  What  is  the  implementation  sequence  of  the  quantitative  and  qualitative  data 
collection  in  the  proposed  study? 

2.  Is  priority  given  to  the  quantitative  or  qualitative  data  collection  and  analysis? 

3.  At  what  stage  in  the  research  project  is  the  quantitative  and  qualitative  data  and 
findings  integrated? 

4.  Was  an  overall  theoretical  perspective  used  in  the  study? 

As  seen  in  Figure  6  below,  the  answers  to  the  questions  will  form  a  “path”  across 
the  decision  matrix.  This  “path”  then  detennines  the  appropriate  strategy.  This  section 
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explains  how  this  decision  tool  was  applied  to  surmise  which  mixed  method  research 
strategy  to  employ. 


Question  1: 
Implementation 

Question  2: 
Priority 

Question  3: 
Integration 

Question  4: 
Theoretical 
Perspective 

No  Sequence 
Concurrent 

Equal 

At  Data  Collection 

Explicit 

Sequential- 
Qualitative  first 

Qualitative 

At  Data  Analysis 

At  Data 
Interpretation 

Sequential- 
Qualitative  first 

Quantitative 

Implicit 

With  Some 
Combination 

Figure  6.  Decision  Matrix  for  Determining  Research  Strategy 


Question  1:  Implementation  Sequence 

The  data  was  collected  in  two  distinct  phases.  First,  the  qualitative  analysis  was 
completed  to  explore  the  topic.  Once  the  issues  were  identified,  a  further  understanding 
of  which  issues  were  most  relevant  was  completed  through  a  quantitative  analysis. 
Therefore,  the  implementation  occurred  sequentially,  from  qualitative  to  quantitative. 

Question  2:  Priority 

According  to  Creswell,  the  second  question  detennines  “whether  greater  priority 
was  given  to  the  qualitative  or  quantitative  approach”  (Creswell,  2003).  Before  assigning 
the  priority  a  research  paradigm  had  to  be  detennined.  The  initial  framework  for  the 
research  strategy  was  established  by  examining  what  this  study  aims  to  discover 
(Titscher,  Meyer  et  al.,  2000).  There  is  no  preexisting  body  of  knowledge  discussing 
how  an  enterprise  architecture  should  be  managed;  therefore,  an  exploratory  research 
strategy  was  selected. 


26 


Since  the  research  paradigm  warrants  an  exploratory  effort,  this  study  requires 
interpretative  procedures  whose  goal  is  the  clarification  of  ideas  or  concepts  and/or  the 
development  of  theoretical  assumptions  (Titscher,  Meyer  et  ah,  2000).  Leedy  and 
Ormrod  state  a  qualitative  approach  should  be  selected  when  developing  new  insight  or 
perspective  about  a  phenomenon  (Leedy  and  Ormrod,  2001).  Therefore,  the  priority  was 
placed  on  the  qualitative  approach. 

Question  3:  Integration 

Even  though  the  priority  was  placed  on  a  qualitative  approach,  one  of  the 
purposes  of  this  research  effort  was  to  identify  which  issues  are  the  most  relevant.  To 
achieve  this  objective  the  qualitative  findings  are  quantitatively  analyzed  and  interpreted 
in  Chapter  4.  Given  that  the  two  phases  do  not  overlap,  support  is  given  to  the  decision 
to  implement  a  sequential  data  collection  methodology  (Creswell,  2003). 

Question  4:  Theoretical  Perspective 

The  literature  review  described  reports  from  the  United  States  General 
Accounting  Office  who  has  developed  an  enterprise  architecture  Capability  Maturity 
Management  Module  (Schekkerman,  2001;  General  Accounting  Office,  2003). 

However,  an  exhaustive  review  of  the  top  ten  journals  from  the  Management  Infonnation 
Systems  field  of  study  did  not  identify  a  theoretical  framework  supporting  this  module. 
Neuendorf  states  (Neuendorf,  2002): 

When  existing  theory  or  research  literature  cannot  give  a  complete  picture  of  the 
message  pool,  the  researcher  may  take  a  more  practical  approach.  The  researcher 
may  need  to  immerse  himself  or  herself  in  the  world  of  the  message  pool  and  conduct 
a  qualitative  scrutiny  of  a  representative  subset  of  the  content  to  be  examined.  In  this 
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way,  variables  emerge  from  the  message  pool,  and  the  investigator  is  well  grounded 
in  the  reality  of  the  messages.  Quite  simply,  the  researcher  needs  to  go  native. 

From  Neuendorfs  recommendation’s  an  explicit  theoretical  framework  was  not 
used  to  guide  this  research  effort.  Instead,  a  theoretical  “lens”  assisted  in  determining 
what  issues  were  important  to  examine  (Creswell,  2003).  From  this  “lens”,  an  inductive 
procedure  was  followed  to  create  an  emergent  model  that  could  be  used  to  identify  the 
key  issues  in  effectively  managing  an  enterprise  architecture.  Therefore,  the  mixed 
method  used  an  implicit  theory  with  the  intent  to  allow  the  issues  to  emerge  from  the 
selected  data  set. 

The  four  questions  were  answered  by  scrutinizing  the  objective  of  this  research. 
Figure  7  summarizes  these  answers  by  shading  in  the  cells  for  each  response.  These  cells 
form  a  “path”  leading  to  the  selection  of  the  appropriate  research  strategy. 


Question  1: 
Implementation 

Question  2: 
Priority 

Question  3: 
Integration 

Question  4: 
Theoretical 
Perspective 

No  Sequence 
Concurrent 

Equal 

At  Data  Collection 

Explicit 

Sequential- 
Qualitative  first 

Qualitative 

At  Data  Analysis 

At  Data 
Interpretation 

Sequential- 
Qualitative  first 

Quantiative 

Implicit 

With  Some 
Combination 

Figure  7.  Decision  Choices  for  Determining  Research  Strategy 


The  “path”  created  requires  qualitative  data  to  be  present  prior  to  the  quantitative 
analysis.  In  addition,  the  main  focus  of  the  research  is  identifying  the  qualitative  themes 
present  in  the  data.  Only  after  these  themes  are  identified  are  they  then  interpreted. 
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Finally,  the  research  paradigm  is  exploratory  in  nature  since  no  preexisting  theories  are 
present  in  the  current  body  of  knowledge. 

Answering  these  four  questions  allowed  the  researcher  to  focus  on  the  necessary 
criteria  to  detennine  the  appropriate  research  strategy.  Furthermore,  from  this  strategy  a 
methodology  is  laid  out  to  achieve  the  objective  of  this  research.  Figure  8  was  created  by 
drawing  on  each  individual  answer  in  the  decision  matrix  to  determine  the  required 
“path”  of  events  to  conduct  this  research. 

Qualitative  Qualitative  Quantitative  Quantitative  Interpretation 

Data  ->  Data  ->  Data  ->  Data  ->  of  Data 
Collection  Analysis  Collection  Analysis  Analysis 

Figure  8.  Research  Method  Strategy 

This  “path”  leads  directly  to  the  utilization  of  a  sequential  exploratory  research  strategy 
(Creswell,  2003).  The  remaining  sections  explain  the  methodology  utilized  to  carry  out 
the  selected  research  strategy. 


Methodology 

The  data  for  this  research  originates  from  written  text  discussing  the  concepts  of 
enterprise  architecture,  enterprise  infrastructure,  systems  development,  and  strategic  data 
planning.  Denzin  and  Lincoln  suggests  that  a  content  analysis  is  an  acceptable  research 
methodology  for  this  type  of  data  (Denzin  and  Lincoln,  2000).  Leedy  and  Ormrod  agree 
that  a  content  analysis  is  the  systematic  examination  of  written  documents  “for  the 
purpose  of  identifying  patterns,  themes,  or  biases”  (Leedy  and  Ormrod,  2001). 

Therefore,  to  carry  out  this  research’s  methodology  a  content  analysis  was  perfonned. 
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In  conducting  the  content  analysis,  a  prescribed  process  was  followed.  First,  the 
content  characteristics  to  be  measured  were  specified.  Then  rules  were  established  to 
identify  and  record  the  characteristics  of  interest.  Finally,  a  quantitative  statistical 
analysis  was  carried  out  to  detennine  if  the  findings  converge  or  diverge  on  these 
characteristics.  The  researcher  employed  Neuendorfs  nine-step  framework  to  carry  out 
this  process  (Neuendorf,  2002).  The  sequence  of  steps  prescribed  by  Neuendorfs 
framework  were  modified  to  reflect  the  actions  taken  by  the  researcher.  By  explicitly 
explaining  how  the  content  analysis  was  conducted,  future  academic  studies  will  be  able 
to  accurately  replicate  the  study.  The  steps  were: 

1)  Theory  and  Rationale 

2)  Sampling 

3)  Conceptualization  Decisions 

4)  Coding  Schemes 

5)  Operationalization  Measures 

6)  Training  and  Initial  Reliability 

7)  Coding 

8)  Final  Reliability 

9)  Tabulation  and  Reporting 
Step  1:  Theory  and  Rationale 

To  generate  data  sources  for  this  research  effort  two  separate  steps  where  taken  to 
identity  relevant  literature  in  the  fields  of  enterprise  architecture,  enterprise  infrastructure, 
systems  development,  and  strategic  data  planning.  This  section  explains  what  sources  of 
literature  were  selected  and  answers  the  question:  Why  were  they  selected? 
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Developing  an  Infonnation  Architecture  has  been  identified  as  one  of  the  top  ten 
management  issues  by  senior  IS  executives  since  1987  (Brancheau  and  Wetherbe,  1987; 
Niederman,  Brancheau  et  al.,  1991;  Brancheau,  Janz  et  ah,  1996).  Nord  and  Nord 
identified  Communications  of  the  ACM,  Decision  Science,  Information  and  Management, 
Information  Systems  Management  (changed  name  from  Journal  of  Information  Systems 
Management),  Journal  of  Computer  Information  Systems,  Journal  of  Management 
Information  Systems,  Journal  of  Systems  Management,  Management  Science,  and  MIS 
Quarterly  as  journals  that  are  considered  to  publish  important  research  (Nord  and  Nord, 
1995).  Information  Systems  Research  was  added  to  this  list  because  it  was  recognized  by 
two  other  research  efforts  as  a  “top  tier”  journal  (Walstrom,  Hardgrave  et  ah,  1995; 
Hardgrave  and  Walstrom,  1997). 

In  the  first  step  taken  to  identify  relevant  literature  each  top  tier  journal  was 
reviewed.  Nord  and  Nord  state  these  academic  journals  publish  important  research; 
therefore,  the  journals  should  contain  articles  covering  the  top  management  issues.  As 
stated  above,  developing  an  architecture  was  one  of  the  top  management  issues  from 
1987  to  the  present  day.  Therefore,  the  title  and  abstract  of  each  article  published  from 
1987  to  August  2004  by  these  ten  academic  journals  was  reviewed.  If  the  research  focus 
of  the  individual  article  dealt  with  enterprise  architecture,  enterprise  infrastructure,  and/or 
strategic  planning  it  was  included  in  the  content  analysis. 

Step  2:  Sampling 

During  the  first  round  of  reviews  only  12  research  studies  were  identified  that 
dealt  with  enterprise  architecture  and/or  enterprise  infrastructure.  Due  to  the  minimal 
amount  of  articles  identified  during  the  initial  review,  this  step  was  repeated.  However, 
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during  the  second  search  of  the  same  set  of  academic  journals  articles  were  included  in 
the  content  analysis  if  the  research  focus  dealt  with  strategic  planning,  IS  planning, 
system  integration,  systems  development  and/or  strategic  use  of  information  systems. 

Nord  and  Nord  reported  there  was  the  potential  for  significant  bias  in  establishing 
which  journals  should  be  rated  as  “top  tier”  (Nord  and  Nord,  1995).  In  addition,  there 
were  a  limited  number  of  articles  identified  during  the  first  review  of  the  top  tier  IS 
management  journals.  Consequently,  the  key  words  from  the  academic  articles  identified 
in  the  first  review  were  used  to  search  for  additional  articles.  The  following  keywords 
were  used  in  the  search  parameters  of  an  academic  library  search  engine  and  various 
online  sources:  enterprise  architecture,  infrastructure,  IS  planning,  infonnation 
technology  strategy,  and  system  integration. 

This  second  step  randomly  designated  academic  articles,  government  and 
commercial  reports,  and  white  papers  for  possible  inclusion  in  the  content  analysis.  Any 
article  dealing  with  enterprise  architecture,  enterprise  infrastructure,  strategic  planning,  IS 
planning,  system  integration,  and  strategic  use  of  information  systems  were  added  as  a 
data  source.  The  expanded  review  of  the  top  IS  management  journals  and  the  online 
search  resulted  in  identifying  exactly  100  more  articles. 

In  total,  the  separate  searchers  allowed  the  primary  researcher  to  identify  112 
articles  for  possible  inclusion  in  the  content  analysis.  Each  article’s  title,  abstract, 
introduction,  and  conclusion  was  reviewed  to  determine  if  the  data  source  should  be 
retained  for  the  study.  This  review  paired  down  the  total  number  of  articles  used  during 
the  content  analysis  from  1 12  to  52. 
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Step  3:  Conceptualization  Decisions 

As  stated  in  Chapter  2’s  literature  review,  there  is  no  existing  theory  stating  how 
an  enterprise  architecture  should  be  managed.  As  a  result,  there  was  no  explicit  theory 
used  in  the  selection  of  the  content  analysis  variables.  However,  the  literature  review 
recognized  Zachman’s  development  of  an  enterprise  architecture  framework.  This 
framework  focused  on  four  key  areas  to  include:  process,  data,  control,  and  technology 
(Zachman,  1987).  Therefore,  these  four  categories  were  used  as  a  theoretical  perspective 
to  identify  management  issues  throughout  the  body  of  knowledge.  How  the  issues  were 
identified  and  validated  is  discussed  in  the  next  section. 

Step  4:  Coding  Schemes 

An  emergent  process  of  variable  identification  was  employed  to  identify  the 
issues  that  must  be  addressed  by  the  Air  Force  to  effectively  manage  its  enterprise 
architecture.  As  each  article  was  read  by  the  primary  researcher,  the  major  issues  were 
noted  on  a  separate  sheet  of  paper.  The  section  of  the  article  from  which  the  issue  was 
extracted  was  highlighted  and  a  comment  was  inserted  into  the  margin  of  the  article. 

This  enabled  the  researcher  to  match  the  notes  to  the  context  of  the  article  if  required 
while  developing  the  codebook  or  during  the  process  of  coding  the  written  text.  This  was 
completed  for  all  52  articles. 

In  utilizing  the  emergent  process  of  variable  identification,  36  separate  issues 
were  recognized  during  the  review  of  the  52  articles.  By  utilizing  the  theoretical 
perspective  of  Zachman’s  Framework,  four  main  categories  were  created  in  the  codebook 
to  include:  process,  data,  control,  and  technology.  Due  to  the  significant  number  of 
issues  under  the  control  category  this  was  further  broken  down  into  three  separate 
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categories  which  are  development  control,  operational  control,  and  maintenance  control. 
Finally,  two  additional  categories  emerged  while  reviewing  the  notes  created  by  the 
primary  researcher.  These  two  categories  are  flexibility  and  openness.  A  codebook  was 
then  developed  by  placing  issues  under  the  category  to  which  it  was  related.  The  primary 
researcher  then  reviewed  each  article’s  highlighted  sections  and  the  corresponding  notes 
to  make  sure  the  issues  were  not  taken  out  of  context  in  developing  the  codebook.  The 
codebook  matrix  is  located  in  Appendix  B :  Codebook. 

Since  identifying  thematic  units  requires  significant  interpretation  to  code 
properly  (Lacity  and  Janson,  1994),  four  co-researchers  were  each  assigned  a  subset  of 
the  52  articles  to  address  the  potential  for  personal  bias  and  to  prevent  errors  of  judgment 
and  misinterpretation  of  the  text.  The  co-researchers  were  instructed  to  review  each 
article  to  ensure  the  primary  researcher  had  not  misinterpreted  the  identified  issues.  If 
there  was  disagreement  between  the  primary  researcher  and  co-researcher  the  section  of 
the  article  was  read  together  and  the  issue  was  discussed  until  they  agreed  upon  the 
interpretation  of  the  issue.  The  co-researcher  was  also  directed  to  review  each  of  the 
issues  listed  in  the  codebook  to  make  sure  the  issue  was  clear  and  concise  to  avoid  any 
potential  misinterpretations  during  the  content  analysis.  Any  pertinent  recommendations 
for  improving  the  codebook  were  incorporated  by  the  primary  researcher.  Other  than 
syntax  corrections,  the  only  major  modification  made  was  combining  two  issues  that 
proved  to  be  redundant  bringing  the  total  number  of  emergent  issues  to  35. 

The  four  co-researchers  all  had  the  same  operational  and  academic  experience  as 
the  primary  researcher.  Just  as  the  primary  research,  all  four  co-researchers  are 
Information  Resource  Management  students  enrolled  at  the  Air  Force  Institute  of 
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Technology.  Furthermore,  both  the  primary  researcher  and  each  of  the  selected  co¬ 
researchers  are  1st  Lieutenants  in  the  Air  Force  and  have  held  one  prior  assignment  as  a 
Communications  and  Information  Officer  before  being  assigned  as  a  Master’s  Degree 
Student  at  Wright  Patterson  Air  Force  Base,  Ohio. 

Step  5:  Operationalization  measures 

The  unit  of  analysis  is  each  of  the  identified  35  issues.  However,  each  of  the 
identified  issues  did  not  appear  in  every  article.  Instead,  if  the  issue  was  determined  to  be 
present  while  reviewing  the  article  a  number  one  was  marked  on  the  codebook  matrix. 

No  mark  was  made  on  the  matrix  if  the  issue  was  not  present  in  the  article.  No  weight 
was  added  due  to  the  author’s  level  of  expertise  or  the  source  of  the  document  being 
reviewed.  The  codebook  matrix  provided  to  each  coder  is  referenced  in  Appendix  B: 
Codebook. 

Step  6:  Training  and  initial  reliability 

Twenty-four  coders  analyzed  the  articles  selected  for  the  content  analysis.  The 
coders  were  enrolled  in  the  Enterprise  Architecture  class  at  the  Air  Force  Institute  of 
Technology.  All  of  the  subjects  are  pursuing  their  master’s  degree  in  the  Systems  and 
Engineering  Management  Department. 

To  prepare  the  coders  for  the  content  analysis  a  training  session  was  held  to 
ensure  they  understood  how  to  use  the  codebook  and  how  to  identify  the  issues  contained 
in  their  assigned  articles.  The  primary  researcher  did  not  explain  the  purpose  of  the  study 
to  the  coders  to  reduce  the  bias  that  would  compromise  validity  (Neuendorf,  2002).  A 
subsample  of  three  articles  was  selected  from  the  pool  of  documents  to  run  a  pilot  test 
with  the  24  coders.  An  assessment  of  the  coders  was  carried  out  to  develop  a  “valid, 
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reliable,  and  useful  coding  scheme”  by  considering  three  diagnostic  measures 
(Neuendorf,  2002). 

1 .  The  identification  of  problematic  measures 

2.  The  identification  of  problematic  categories 

3.  The  identification  of  problematic  coders 

After  the  pilot  study  was  completed,  each  of  the  measures  listed  above  was 
assessed.  This  was  completed  to  determine  if  changes  in  the  codebook  was  necessary 
and/or  if  the  coders  required  addition  training.  This  technique  was  used  to  establish 
intercoder  reliability  by  ensuring  all  24  coders  have  the  same  understanding  of  the  coding 
scheme  (Neuendorf,  2002).  The  results  and  interpretation  of  the  pilot  study  are  reported 
in  Chapter  4. 

Step  7:  Coding 

The  primary  researcher  coded  all  of  the  articles  included  in  the  content  analysis 
by  recording  the  results  on  the  codebook  matrix.  As  stated  above,  to  reduce  the  influence 
of  personal  bias  four  co-researchers  validated  these  findings  for  their  assigned  subset  of 
articles. 

The  coders  consisted  of  students  enrolled  in  the  Enterprise  Architecture  course  for 
the  Fall  2004  quarter.  From  these  24  students  two  groups  of  12  were  established  to 
determine  if  educational  or  vocational  experience  confounded  the  outcome  of  the  results. 
The  first  group  of  12  students  consisted  of  1 1  Majors  enrolled  in  the  Intennediate 
Development  Education  (IDE)  program  and  one  Captain  in  his  second  to  last  quarter 
before  graduation.  The  second  group  comprised  12  students  who  were  in  their  first 
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quarter  of  classes  in  pursuit  of  their  Master’s  degree  in  the  Systems  and  Engineering 
Management  Department. 

Each  of  the  24  coders  was  assigned  one  identical  article  to  measure  for  inter-coder 
reliability  (Neuendorf,  2002).  Then  the  two  groups  of  twelve  were  each  broken  down 
into  groups  of  two  students  each  to  create  a  total  of  12  groups.  One  student  from  the  IDE 
program  was  assigned  the  exact  same  articles  as  one  student  from  the  Information 
Resource  Management  program.  This  created  twelve  paired  groups  between  the  two 
programs  of  study. 

Each  of  the  twelve  groups  were  assigned  a  total  of  five  articles.  Four  articles 
were  randomly  assigned  to  each  member  of  each  group  and  the  fifth  article  assigned  to 
every  coder  served  as  the  required  overlap  as  discussed  above.  Once  the  articles  were 
assigned  to  each  member  within  the  groups,  the  individual  coders  were  instructed  to 
annotate  their  results  on  the  codebook  that  had  been  provided. 

Step  8:  Final  Reliability 

Weber  asserts:  “To  make  valid  inferences  from  the  text,  it  is  important  that  the 
classification  procedure  be  reliable  in  the  sense  of  being  consistent:  different  people 
should  code  the  same  text  in  the  same  way.”  Weber  continues  to  discuss  the  issue  of 
reliability  by  stating  “problems  usually  grow  out  of  the  ambiguity  of  word  meanings, 
category  definitions,  or  other  coding  rules”(Weber,  1990).  The  following  two  types  of 
reliability  are  pertinent  to  content  analysis: 

a.  Stability  -  Addresses  how  consistent  the  results  of  the  content  classification  are 
over  time. 
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b.  Reproducibility  (inter-coder  consistency)  -  Determines  if  the  content 
classification  produces  matching  results  when  the  identical  text  is  coded  by  more  than 
one  person. 

As  mentioned  in  Step  7,  the  24  coders  were  divided  into  two  groups.  Then  one 
coder  from  each  group  was  randomly  paired  up  with  a  coder  from  the  other  group.  Each 
pair  of  coders  was  then  assigned  the  exact  same  five  articles  to  analyze.  To  confirm  the 
stability  of  the  coding  schema  an  independent  t-tests  was  perfonned  to  detennine  if  the 
average  percent  agreement  for  each  group  of  two  coders  was  or  was  not  statistically 
different. 

Reproducibility  is  addressed  by  measuring  the  agreement  between  each  of  the 
coders  and  the  primary  researcher.  The  use  of  the  appropriate  reliability  coefficient 
calculation  is  important.  However,  if  the  coders  are  consistently  making  incorrect 
judgments  about  the  presence  or  absence  of  the  issues  in  the  article  being  coded  the  level 
of  reproducibility  will  be  negatively  affected  (Kolbe  and  Burnett,  1991).  The  primary 
researcher  improved  the  reproducibility  of  this  research  by  placing  emphasis  on 
improving  the  operational  procedures  used  to  properly  code  the  content  analysis  articles. 
Focusing  on  the  underlying  classifications  scheme,  the  operational  definitions  for  coding 
categories,  and  the  directions  that  guide  the  coding  process  directly  improves  the  quality 
of  judgment-based  data  (Perreault  and  Leigh,  1989). 

To  measure  the  strength  of  the  research  method  employed  a  coefficient  of 
agreement  calculation  was  completed.  The  coefficient  most  commonly  used  in  content 
analysis  due  to  its  applicability  and  ease  of  use  is  percent  agreement  (Perreault  and  Leigh, 
1989;  Kolbe  and  Burnett,  1991;  Neuendorf,  2002).  Conversely,  this  coefficient  has  been 
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identified  as  having  the  potential  to  over-inflate  the  level  of  agreement  due  to  “chance 
agreement”  (Neuendorf,  2002).  Chance  agreement  is  directly  impacted  by  the  number  of 
coding  decisions.  As  the  number  of  issues  in  the  codebook  increases  the  probability  of 
chance  agreement  decreases  (Perreault  and  Leigh,  1989;  Kolbe  and  Burnett,  1991). 

Since  this  research  had  35  issues,  chance  agreement  was  not  seen  as  a 
confounding  factor.  Therefore,  percent  agreement  was  selected  as  the  inter-rater 
reliability  coefficient.  An  agreement  is  defined  as  the  two  judges,  the  primary  researcher 
and  the  coder,  found  the  issue  in  the  article  or  if  both  of  them  agreed  the  issue  was  not 
present  in  the  article.  For  both  the  pilot  and  full  study  the  percent  agreement  for  each  of 
the  24  coders  was  calculated  twice. 

First,  the  coder’s  overall  level  of  agreement  with  the  primary  researcher  was 
measured.  This  was  accomplished  by  totaling  the  number  of  agreements  for  each  of  the 
articles  coded  then  dividing  by  the  total  number  of  issues  (36  for  the  pilot  study  and  35 
for  the  full  study).  Then  the  coder’s  percent  agreement  average  was  computed  for  all  the 
articles  coded  (3  articles  for  the  pilot  study  and  5  articles  for  the  full  study).  However, 
according  to  Neuendorf,  reliability  coefficients  must  be  reported  separately  for  each  and 
every  measured  variable  (Neuendorf,  2002).  Therefore,  the  second  percent  agreement 
measurement  calculated  the  coder’s  level  of  agreement  for  each  issue.  To  calculate  this 
figure  the  total  number  of  agreements  was  divided  by  the  number  of  articles  coded.  Once 
again  the  coder’s  percent  agreement  average  was  computed  across  all  of  the  issues. 

Each  coder’s  two  measurements  of  percent  agreement  were  then  plotted  on  a 
separate  histogram.  These  two  distributions  allowed  the  researcher  to  calculate  a 
confidence  interval  for  the  computed  level  of  percent  agreement.  From  these  two 
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confidence  intervals,  the  overall  reliability  between  the  judges  was  established  allowing 
the  primary  researcher  to  make  inferences  about  the  results. 

Step  9:  Tabulation  and  Reporting 

The  final  step  of  Neuendorf  s  framework  determined  which  of  the  35  emergent 
issues  were  the  most  relevant.  To  make  this  detennination  the  primary  researcher 
calculated  the  overall  level  of  presence  of  each  of  the  identified  themes  across  the  49 
articles  included  in  the  content  analysis.  The  themes  with  the  highest  frequency  levels 
were  detennined  to  be  the  most  relevant  issues  the  Air  Force  must  focus  on  to 
successfully  manage  its  enterprise  architecture.  This  answered  the  fourth  investigative 
question  of  this  research: 

4.  Which  issues  have  the  most  relevance? 

The  identified  issues  were  then  further  analyzed  to  identify  trends  and  consistent 
themes.  This  analysis  answered  the  fifth  investigative  question: 

5.  What  does  the  literature  identify  as  the  underlying  factors  driving  these  issues? 

Research  Strategy  Limitations 

The  selected  research  strategy  introduced  some  limiting  factors  that  may  affect 
the  results  of  this  study.  In  a  content  analysis  the  researcher  is  a  key  instrument  (Leedy, 
2001).  This  causes  the  results  to  be  confounded  by  the  inescapable  human  nature  of  the 
researcher.  For  example,  the  article  analysis  is  inherently  subjective.  Therefore,  the 
results  may  be  impacted  by  external  variables. 

Another  limitation  is  the  content  analysis  selectively  identified  latent  issues.  This 
allowed  the  subjective  judgment  of  the  primary  researcher  to  be  introduced  in  the 
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development  of  the  coding  schema.  Therefore,  the  schema  may  not  be  representative  of 
the  identified  articles,  limiting  the  ability  to  generalize  about  the  findings  to  a  larger 
population  of  articles. 

Summary 

A  methodical  approach  was  taken  in  choosing  the  appropriate  research  strategy 
required  to  answer  the  five  investigative  question  of  this  research.  By  selecting  a 
sequential  exploratory  research  strategy,  quantitative  data  was  used  to  interpret  the 
qualitative  findings  from  the  content  analysis.  This  enabled  the  researcher  to  identify  the 
most  relevant  issues  the  Air  Force  must  focus  on  to  effectively  manage  its  enterprise 
architecture. 
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IV.  Results  and  Analysis 


Overview 

The  purpose  of  this  chapter  is  to  present  the  answers  to  the  following  five 
investigative  questions: 

1 .  What  is  an  enterprise  architecture? 

2.  How  does  the  Air  Force  define  its  enterprise  architecture? 

3.  What  does  the  literature  identify  as  the  issues  that  must  be  addressed  to  effectively 
manage  an  enterprise  architecture? 

4.  Which  issues  have  the  most  relevance? 

5.  What  does  the  literature  identify  as  the  underlying  factors  driving  these  issues? 
The  first  two  questions  will  be  answered  by  recounting  the  operational  definitions 

as  expounded  upon  in  Chapter  2’s  literature  review.  Then  a  description  of  the  analyzed 
articles  is  presented.  Prior  to  presenting  the  results  of  the  content  analysis  the 
measurement  instrument’s  validity  and  reliability  is  established  by  presenting  the  results 
from  both  the  pilot  and  the  full  study.  The  instrument  is  then  utilized  to  answer  the  final 
three  investigative  questions.  Finally,  the  answers  to  these  questions  are  discussed  to 
reach  the  overall  goal  of  this  research. 

Operational  Definitions 

In  Chapter  2,  the  researcher  provided  an  account  of  the  development  of  the 
enterprise  architecture  concept.  This  was  followed  up  with  a  historical  account  of  the 
implementation  of  both  the  Department  of  Defense’s  and  the  Air  Force’s  enterprise 
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architecture.  Providing  this  context  served  as  the  necessary  background  information  to 
formulate  answers  to  the  first  two  investigative  questions: 

1 .  What  is  an  enterprise  architecture? 

2.  How  does  the  Air  Force  define  its  enterprise  architecture? 

To  answer  both  of  these  questions  the  researcher  gathered  evidence,  determined 
patterns,  and  then  develop  an  agreed  upon  operational  definition  for  each  question 
(Mason,  McKenney  et  ah,  1997).  First,  through  the  review  of  academic  literature  the 
researcher  was  able  to  identify  common  themes  resulting  in  the  following  operational 
definition  of  an  enterprise  architecture: 

The  organization  of  computing  resources  in  an  organization,  which  consists  of  data, 
information,  applications,  infrastructure,  and  personnel  to  enable  a  firm’s  business 
strategy. 

The  purpose  of  this  research  was  to  assist  the  Air  Force  Communications  Agency 
by  identifying  the  key  issues  to  effectively  manage  the  Air  Force’s  Enterprise 
Architecture.  Therefore,  the  next  logical  step  was  to  identify  an  operational  definition  for 
the  Air  Force’s  Enterprise  Architecture.  A  description  of  how  the  Air  Force’s  Enterprise 
Architecture  Framework  was  created  was  presented  in  Chapter  2.  From  this  description 
the  researcher  was  able  to  present  the  following  operational  definition  for  the  Air  Force’s 
Enterprise  Architecture: 

The  Air  Force  Enterprise  Architecture  is  a  tool  that  allows  the  vision,  mission,  and 
operational  planning  to  be  synchronized  with  programming,  budgeting,  and 
acquisition  management  to  achieve  the  Air  Force’s  strategic  objectives. 
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Sample 

The  data  set  for  this  research  consisted  of  52  articles.  Among  these  articles  19 
originated  in  practitioner  journals,  two  in  government  reports,  8  in  white  papers,  and  the 
remaining  23  appeared  in  peer-reviewed  academic  journals.  The  peer-reviewed  articles 
consisted  of  nine  case  studies,  four  comparative  analysis,  three  developmental  studies, 
three  surveys,  and  three  Delphi  studies. 

Validity  of  Measurement  Instrument 

To  increase  the  level  of  objectivity  of  the  coding  process  the  primary  researcher 
addressed  shortfalls  in  the  creation  and  operationalization  of  the  measurement  instrument. 
Once  the  initial  coding  schema  had  been  created  an  independent  review  was  completed  to 
remove  the  primary  researcher’s  personal  bias.  In  addition,  the  24  coders  completed  a 
pilot  study  to  detennine  if  the  coding  schema  and/or  provided  directions  required  any 
modifications.  The  following  two  sections  explain  the  results  and  steps  taken  to  improve 
upon  the  measurement  instrument’s  validity. 

Issue  Validation 

As  stated  in  Chapter  3,  an  emergent  process  of  variable  identification  was 
employed  by  the  primary  researcher  to  identify  the  issues  that  must  be  addressed  to 
manage  an  enterprise  architecture.  The  emergent  process  of  variable  identification 
resulted  in  36  separate  issues  to  be  recognized  during  the  review  of  the  52  articles. 

This  process  does  not  allow  the  intricacies  of  human  nature  to  be  removed  leading  to 
personal  bias,  errors  of  judgment,  and  misinterpretation  of  the  text. 


44 


Therefore,  four  co-researchers  were  each  assigned  a  subset  of  the  52  articles  to 
address  the  potential  for  misinterpreted  the  identified  issues.  Each  co-researcher  was 
given  13  articles  to  review.  Amongst  the  four  co-researchers,  only  two  of  them  disagreed 
with  the  primary  researcher  in  regards  to  the  presence  of  an  issue  within  an  article. 

The  first  of  these  two  co-researchers  had  identified  four  separate  articles  with  only 
one  disagreement  per  article.  The  primary  researcher  and  the  co-researcher  discussed 
each  individual  disagreement  and  on  three  of  the  them  the  two  individuals  came  to 
agreement  concerning  the  proper  interpretation  of  the  text.  These  agreements  led  to  the 
primary  researcher  removing  the  annotation  that  the  issue  was  present  in  the  article  in 
question  for  each  of  the  three  articles.  On  the  fourth  article,  both  researchers  agreed  the 
issue  was  present  leading  to  making  no  changes  to  the  annotation  on  the  coding  schema 
that  the  issue  was  present. 

The  other  co-researcher  who  disagreed  with  the  interpretations  of  the  primary 
researcher  disputed  three  issues  spanning  two  articles.  Once  again  the  two  researchers 
discussed  the  disagreements  leading  to  an  agreement  that  the  issues  were  present  in  the 
articles.  This  resulted  in  making  no  changes  to  the  annotation  on  the  primary 
researcher’s  coding  schema. 

The  four  co-researchers  also  reviewed  the  coding  schema  to  check  for  syntax  or 
spelling  errors  and  to  ensure  there  was  no  redundancy  across  the  36  issues.  One  of  the 
co-researchers  recommended  combining  two  issues  in  the  coding  schema  to  reduce  the 
possibility  of  misinterpretation  during  the  content  analysis.  The  two  issues  that  were 
combined  into  one  are  the  following: 
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-  An  interoperable  architecture  allows  other  systems  to  easily  integrate  into  it— 
systems  can  be  moved  in  and  out  of  the  architecture 

-  Avoid  intrusive  integration  by  modifying  code  in  legacy  systems— use  data  brokers 
to  transform  data  from  one  format  to  another 

These  second  issue  was  merged  into  the  first  resulting  in  one  combined  issue: 

-  An  interoperable  architecture  allows  other  systems  to  seamlessly  integrate  with  it, 
allowing  other  systems  to  be  moved  in  and  out  of  the  architecture 

Any  pertinent  recommendations  for  improving  the  codebook  were  incorporated 
by  the  primary  researcher.  This  ensured  the  issues  listed  in  the  coding  schema  were  clear 
and  concise  to  avoid  any  potential  misinterpretations  during  the  content  analysis.  This 
independent  review  resulted  in  removing  the  personal  bias  of  the  primary  researcher  and 
increasing  the  level  of  objectivity  in  coding  the  content  analysis  articles. 

Instrument  Validation  -  Pilot  Study 

A  sub-sample  consisting  of  three  of  the  52  articles  was  selected  to  conduct  a  pilot 
study.  Each  of  the  24  coders  independently  coded  each  article  included  in  the  sub¬ 
sample.  This  pilot  study  was  conducted  to  develop  a  “valid,  reliable,  and  useful  coding 
schema”  by  considering  three  diagnostic  measures  (Neuendorf,  2002): 

1 .  The  identification  of  problematic  measures 

2.  The  identification  of  problematic  categories 

3.  The  identification  of  problematic  coders 

The  assessment  of  these  three  measures  and  the  action  taken  by  the  primary  researcher  to 
further  develop  the  coding  schema  is  explained  in  the  next  four  sections. 

Diagnostic  1:  Problematic  Measures 
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To  identify  problematic  measures  the  percent  agreement  for  each  article  amongst 
the  24  coders  and  the  primary  researcher  was  computed.  The  overall  percent  agreement 
for  each  article  was  calculated  by  dividing  the  total  number  of  agreements  by  the  total 
number  of  issues.  Then  each  coder’s  average  percent  agreement  amongst  the  three 
articles  was  determined.  The  24  percent  agreement  averages  were  then  inputted  into 
JMP  version  5.1.  This  program  was  utilized  to  produce  the  histogram  and  statistical 
measurements  presented  in  Figure  9  and  Figure  10  below. 


Figure  9.  Pilot  Study  Average  Article  Agreement  per  Coder 


Mean 
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Std  Dev 
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Std  Err  Mean 
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upper  95%  Mean 
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lower  95%  Mean 
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Figure  10.  Pilot  Study  Article  Agreement  Statistical  Measurements 

As  shown  in  Figure  10  above,  the  mean  for  the  coders’  average  percent  agreement 
is  68.02%.  The  distribution  appeared  to  be  normally  distributed  and  the  Shapiro-Wilk 
test,  a  statistical  test  for  normality,  supported  this  claim.  The  Shapiro-Wilk  test  evaluated 
the  following  two  hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distribution  is  normally  distributed 
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Ha:  The  distribution  is  not  normally  distributed 

The  reported  p-value  was  0.7627,  causing  the  primary  researcher  to  fail  to  reject 
the  null  hypothesis,  supporting  the  claim  the  histogram  was  normally  distributed. 
Therefore,  the  confidence  interval  can  be  used  to  infer  the  coders  will  have  an  average 
percent  agreement  between  70.86%  and  65.18%  for  this  sub-sample  of  articles  95%  of 
the  time.  Nine  coders  scored  above  the  upper  bound  and  seven  coders  scored  below  the 
lower  bound.  To  identify  problematic  measures  the  seven  coders  who  scored  below  the 
lower  bound  percent  agreement  for  each  article  was  examined.  Four  of  the  coder’s 
lowest  score  occurred  on  the  third  article,  two  on  the  second  article,  and  one  on  the  first 
article. 

Since  over  57%  of  the  coders  lowest  score  occurred  on  the  third  article  the  mean 
was  calculated  for  all  24  coder’s  assigned  articles.  As  shown  in  Table  3  below,  the  mean 
score  for  Article  3  is  60.80%. 


Table  3.  Pilot  Study  Article  Mean  Scores 


Article  1 

Article  2 

Article  3 

Mean 

0.7202 

Mean 

0.7027 

Mean 

0.6080 

This  score  is  well  below  the  95%  confidence  interval’s  lower  bound.  Each  coder  was 
assigned  the  same  three  articles.  In  addition,  every  article  was  identified  by  the  same 
numbering  system.  Therefore,  the  ability  to  properly  analyze  article  three  was  identified 
as  problematic. 

Diagnostic  2:  Problematic  Categories 

The  second  diagnostic  measure  examined  the  average  percent  agreement  per  issue 
for  each  of  the  24  coders.  This  measure  was  calculated  by  adding  up  the  total  number  of 
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agreements  for  each  issue  across  the  three  articles.  The  sum  was  then  divided  by  the 
number  of  articles  coded.  Then  an  overall  percent  agreement  was  computed  by  averaging 
all  of  the  coders’  respective  scores  per  issue.  The  36  percent  agreement  averages  were 
then  inputted  into  JMP  version  5.1.  This  program  was  utilized  to  produce  the  histogram 
and  statistical  measurements  presented  in  Figure  1 1  and  Figure  12  below. 


Figure  11.  Pilot  Study  Agreement  per  Issue 
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Figure  12.  Pilot  Study  Agreement  per  Issue  Statistical  Measurements 

As  shown  in  Figure  12  above,  the  mean  for  the  overall  percent  agreement  for  all 
the  issues  is  66.44%.  The  distribution  does  not  appear  to  be  normally  distributed  and  the 
Shapiro-Wilk  test  supported  this  claim.  The  Shapiro-Wilk  test  evaluated  the  following 
two  hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distribution  is  normally  distributed 
Ha:  The  distribution  is  not  normally  distributed 
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The  reported  p-value  was  0.0101,  causing  the  primary  researcher  to  reject  the  null 
hypothesis,  supporting  the  claim  the  histogram  was  not  normally  distributed.  Therefore, 
the  confidence  interval  could  not  be  used  to  make  inferences  about  the  issues’  average 
percent  agreement.  However,  the  histogram  appears  to  have  two  separate  distributions 
contained  within  it.  The  distribution  located  above  the  mean  consisted  of  24  issues.  The 
other  distribution  contained  the  remaining  12  issues.  These  12  issues’  average  percent 
agreements  were  all  below  the  lower  bound  of  the  confidence  interval.  Thus,  these  12 
issues  were  identified  as  problematic. 

Diagnostic  3:  Problematic  Coders 

The  identification  of  problematic  coders  was  accomplished  by  re-analyzing  the 
measurement  utilized  to  examine  problematic  issues.  Instead  of  inspecting  the  percent 
agreement  for  one  issue  across  all  the  coders,  the  average  percent  agreement  was 
calculated  across  all  the  issues  for  each  coder.  This  measurement  allowed  the  primary 
researcher  to  identify  any  potential  rogue  coders.  The  percent  agreement  per  issue  for 
each  coder  had  already  been  calculated  during  the  previous  diagnostic  measurement. 

Thus,  the  average  percent  agreement  per  coder  was  calculated  by  adding  up  each  coder’s 
percent  agreement  scores  for  each  issue  and  then  dividing  this  sum  by  36.  The  24  percent 
agreement  averages  were  then  inputted  into  JMP  version  5.1 .  This  program  was  utilized 
to  produce  the  histogram  and  statistical  measurements  presented  in  Figure  13  and  Figure 
14  below. 
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Figure  13.  Pilot  Study  Student  Agreement  Across  Issues 


Mean 

0.6650417) 

^Std  Dev 

0.10461291 

:Std  Err  Mean 

0.0213543 

i  upper  95%  Mean 

0.70921583 

|  lower  95%  Mean 

0.62086751 

N 

24; 

Figure  14.  Pilot  Study  Agreement  Across  Issues  Statistical  Measurements 

As  shown  in  Figure  14  above,  the  mean  percent  agreement  across  all  the  issues  is 
66.5%.  The  distribution  appears  to  be  normally  distributed,  but  the  Shapiro-Wilk  test 
does  not  support  this  claim.  The  Shapiro-Wilk  test  evaluated  the  following  two 
hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distribution  is  normally  distributed 
Ha:  The  distribution  is  not  normally  distributed 

The  reported  p-value  was  0.01 1,  causing  the  primary  researcher  to  reject  the  null 
hypothesis,  supporting  the  claim  the  histogram  was  not  normally  distributed.  An  outlier 
value  of  33.33%  was  excluded  and  the  histogram  and  statistical  measurements  were 
regenerated.  Figure  15  and  Figure  16,  located  below,  present  the  updated  histogram  and 
statistical  measurements  without  the  outlier  measurement. 
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Figure  15.  Pilot  Study  Outlier  Removed 
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Figure  16.  Pilot  Study  Outlier  Removed  Statistical  Measurements 

As  shown  in  Figure  16  above,  the  mean  for  the  coders’  percent  agreement  across 
all  the  issues  is  67.95%.  The  distribution  appears  to  be  normally  distributed  and  the 
Shapiro-Wilk  test  supports  this  claim.  The  Shapiro-Wilk  test  evaluated  the  following  two 
hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distribution  is  normally  distributed 
Ha:  The  distribution  is  not  normally  distributed 

The  reported  p-value  was  0.3386,  causing  the  primary  researcher  to  fail  to  reject 
the  null  hypothesis;  supporting  the  claim  the  histogram  is  normally  distributed.  The 
regenerated  confidence  interval  was  then  used  to  infer  95%  of  the  time  the  remaining  23 
coders’  average  percent  agreement  across  all  the  issues  will  range  between  71.36%  and 
64.54%  for  this  sub-sample  of  articles.  Five  coders  scored  below  the  lower  bound  of  the 
confidence  interval  bringing  the  total  number  of  problematic  coders  to  six. 


52 


Validation  of  Coding  Schema 

These  three  diagnostic  measurements  were  then  holistically  reviewed  to  develop  a 
“valid,  reliable,  and  useful  coding  schema.”  This  review  allowed  the  primary  researcher 
to  make  the  necessary  modifications  to  both  the  measurement  instrument  and  the  coders. 

As  was  noted  in  the  first  and  last  diagnostic  measurements,  there  were  both 
problematic  measures  and  problematic  coders  identified.  To  remedy  these  two  issues  the 
coders  were  given  a  second  training  session  to  explain  how  to  properly  analyze  the 
articles  and  record  their  findings.  Although  the  subjective  judgment  of  the  coders  could 
not  be  entirely  removed  the  repeated  training  was  an  attempt  to  nullify  this  confounding 
factor. 

The  second  diagnostic  measurement  identified  12  problematic  issues.  Each  issue 
was  reviewed  by  the  primary  researcher  to  provide  a  more  clear  and  concise  definition  of 
each  of  the  issues.  In  addition,  the  coders  provided  comments  to  the  primary  researcher 
consisting  of  ways  to  improve  upon  the  coding  schema.  The  pertinent  recommendations 
were  incorporated  into  the  instrument.  The  validated  coding  schema,  located  in 
Appendix  C:  Validated  Codebook,  replaced  the  original  one  for  use  in  the  full  study. 

Reliability  of  Measurement  Instrument  -  Full  Study 

The  validity  of  the  issues  included  in  the  coding  schema  was  established  from  the 
pilot  study.  The  next  step  was  to  confirm  the  reliability  of  operationalizing  the 
measurement  instrument  during  the  analysis  of  the  remaining  49  articles.  As  stated  in 
Chapter  3,  Weber  asserts:  “To  make  valid  inferences  from  the  text,  it  is  important  that 
the  classification  procedure  be  reliable  in  the  sense  of  being  consistent — different  people 
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should  code  the  same  text  in  the  same  way”(Weber,  1990).  The  following  two  types  of 
reliability  are  pertinent  to  content  analysis: 

a.  Stability  -  Addresses  how  consistent  the  results  of  the  content  classification  are 
over  time. 

b.  Reproducibility  (inter-coder  consistency)  -  Determines  if  the  content 
classification  produces  matching  results  when  the  identical  text  is  coded  by  more 
than  one  person. 

The  next  two  sections  describe  the  analysis  of  these  two  types  of  reliability. 

Stability 

Stability  is  established  by  proving  the  same  results  are  obtained  in  a  renewed 
application  of  the  measurement  instrument  to  the  same  text  (Titscher,  Meyer  et  ah,  2000). 
To  confirm  the  stability  of  the  instrument  the  24  coders  were  divided  into  two  groups. 
Then  one  coder  from  each  group  was  randomly  paired  up  with  a  coder  from  the  other 
group.  Each  pair  of  coders  was  then  assigned  the  exact  same  five  articles  to  analyze. 

For  each  coder,  the  average  percent  agreement  for  each  issue  was  calculated 
across  the  five  articles.  This  measure  was  calculated  by  adding  up  the  total  number  of 
agreements  for  each  issue.  The  sum  was  then  divided  by  the  number  of  articles  coded. 

In  addition,  the  overall  percent  agreement  for  each  article  was  calculated  by  dividing  the 
total  number  of  agreements  by  the  total  number  of  issues.  Then  each  coder’s  average 
percent  agreement  amongst  the  five  articles  was  detennined. 

The  two  percent  agreement  measurements  for  the  12  coders  in  both  of  the  groups 
were  then  used  to  create  four  separate  frequency  distributions,  as  shown  in  Figure  17. 
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Figure  17.  Two  Groups’  Percent  Agreement  Frequency  Distributions 


Each  of  these  frequency  distributions  does  not  appear  to  be  normally  distributed. 
However,  the  Shapiro-Wilk  test  was  used  to  evaluate  the  four  distributions  against  the 
following  two  hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distributions  are  normally  distributed 
Ha:  The  distributions  are  not  normally  distributed 
The  p-value  for  each  frequency  distribution  is  presented  in  Table  4. 


Table  4.  Group  Distributions  P-values 


Distribution 

P-Value 

IDE  Issue  Distribution 

0.5642 

IRM  Issue  Distribution 

0.0804 

IDE  Article  Distribution 

0.4259 

IRM  Article  Distribution 

0.6492 

The  p-value  for  each  percent  agreement  measurement  was  above  the  significance 
level  of  0.05.  Therefore,  the  primary  researcher  failed  to  reject  the  null  hypotheses, 
supporting  the  claim  that  each  distribution  is  normally  distributed.  In  addition,  each 
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group’s  population  variances  were  examined  to  ensure  they  were  approximately  equal. 
These  characteristics  allowed  two  independent  t-tests  to  be  perfonned  to  prove  the 
average  percent  agreement  per  issue  and  the  average  percent  agreement  across  the  articles 
for  each  group  of  two  coders  was  not  statistically  different. 

The  first  independent  t-tests  compared  Group  l’s  mean  percent  agreement  per 
issue,  pi,  to  Group  2’s  mean  percent  agreement,  p2.  This  test  evaluated  the  following 
two  hypotheses  at  a  significance  level  of  0.05  with  a  degree  of  freedom  of  44: 

H0:  pi  -  P2  =  0 
Ha-  Pi  "  P2  -f-  0 

The  test  statistic  was  calculated  using  the  formula  indicated  below: 


t  = 


Mi  ~Mi 


0.0311304 


if  i  n 
i  — + — 

- 

r  i 

n 

H - 

{til  n2  ) 

V 

V.23 

23  J 

=  0.4039 


0.0683103 


The  test  statistic  was  between  the  critical  values  of  ta=05/2=  -2.3207  and  2.3207  causing 
the  primary  researcher  to  fail  to  reject  the  null  hypothesis.  Therefore,  the  average  percent 
agreement  per  issue  for  each  group  was  not  statistically  different. 

The  second  independent  t-tests  compared  Group  1  ’s  mean  percent  agreement 
across  the  five  articles,  pi,  to  Group  2’s  mean  percent  agreement,  p2.  This  test  evaluated 
the  following  two  hypotheses  at  a  significance  level  of  0.05  with  a  degree  of  freedom  of 
22: 


H0:  pi  -  p2  =  0 
Ha.  pi  -  P2  -t-  0 

The  test  statistic  was  calculated  using  the  formula  indicated  below: 
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Mi  ~Mi 


0.0311667 


t  = 


0.0.224822 


m  n 

— +  —  0.1153065 

U2  12  J 


The  test  statistic  was  between  the  critical  values  of  ta=05/2  =  -2.4055  and  2.4055  causing 
the  primary  researcher  to  fail  to  reject  the  null  hypothesis.  Therefore,  the  average  percent 
agreements  across  the  five  articles  were  not  statistically  different.  The  results  from  these 
two  tests  establish  the  stability  of  the  coding  schema  by  proving  the  results  are 
statistically  similar  in  a  renewed  application  of  the  measurement  instrument  to  the  same 
text. 

Reproducibility 

The  appropriate  test  to  establish  reproducibility  is  inter-coder  reliability.  To 
perform  this  test  one  article  was  selected  to  be  analyzed  by  all  the  coders.  For  each  issue 
the  total  number  of  agreements  between  the  primary  researcher  and  the  coder  was  added 
up.  This  sum  was  then  divided  by  24,  the  total  number  of  coders.  This  procedure  was 
repeated  for  the  remaining  issues  included  in  the  full  study.  The  average  percent 
agreements  for  all  35  issues  were  inputted  into  JMP  version  5.1  to  produce  the 
histogram  and  statistical  measurements  presented  in  Figure  18  and  Figure  19  below. 
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Figure  18.  Full  Study  Average  Issue  Agreement 
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Mean 

0.5940476 

Std  Dev 
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Std  Err  Mean 
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upper  95%  Mean 
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lower  95%  Mean 

0.5242435 

N 

35 

Figure  19.  Full  Study  Average  Issue  Agreement  Statistical  Measurements 

As  shown  in  Figure  19  above,  the  mean  for  the  coders’  average  percent  agreement 
for  all  the  issues  is  59.40%.  The  distribution  appeared  to  be  normally  distributed  and  the 
Shapiro-Wilk  test  supported  this  claim.  The  Shapiro-Wilk  test  evaluated  the  following 
two  hypotheses  at  a  significance  level  of  0.05: 

H0:  The  distribution  is  normally  distributed 
Ha:  The  distribution  is  not  normally  distributed 

The  reported  p-value  was  0. 1450,  causing  the  primary  researcher  to  fail  to  reject 
the  null  hypothesis,  supporting  the  claim  the  histogram  was  normally  distributed. 
Therefore,  the  confidence  interval  can  be  used  to  infer  that  95%  of  the  time  the  coders 
will  have  an  average  percent  agreement  across  all  the  issues  of  66.39%  to  52.42%. 

The  distribution  was  further  analyzed  to  identify  problematic  issues.  Twelve 
issues  were  identified  below  the  confidence  interval’s  lower  bound.  Therefore,  the 
reliability  of  these  issues  could  not  be  supported  causing  the  primary  researcher  to 
remove  these  issues  from  the  measurement  instrument.  This  brought  the  total  number  of 
issues  down  from  35  to  23  issues.  Figure  20  and  Figure  21,  located  below,  present  the 
updated  histogram  and  statistical  measurements  excluding  the  12  problematic  issues. 
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Figure  20.  Full  Study  Problematic  Issues  Removed 
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Figure  21.  Full  Study  Problematic  Issues  Removed  Statistical  Measurements 

As  shown  in  Figure  21  above,  the  mean  for  the  coders’  average  percent  agreement 
for  the  remaining  issues  increased  from  59.40%  to  70.65%.  The  distribution  does  not 
appear  to  be  normally  distributed  and  the  Shapiro-Wilk  test  supported  this  claim.  The 
Shapiro-Wilk  test  evaluated  the  following  two  hypotheses  at  a  significance  level  of  0.05: 
H0:  The  distribution  is  normally  distributed 
Ha:  The  distribution  is  not  normally  distributed 

The  reported  p-value  was  0.0022,  causing  the  primary  researcher  to  reject  the  null 
hypothesis,  supporting  the  claim  the  histogram  was  not  normally  distributed.  Therefore, 
the  confidence  interval  could  not  be  used  to  make  inferences  about  the  coders’  average 
percent  agreement  for  the  remaining  issues.  However,  the  issues  were  almost  evenly 
distributed  below  and  above  the  mean  with  13  issues  below  and  10  issues  above  the 
mean. 
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To  declare  a  percent  agreement  reliable  there  must  be  at  least  70%  agreement 
(Frey,  Botan  et  al.,  2000).  For  the  10  issues  located  above  the  mean  the  reliability 
coefficient  range  was  between  91.67%  and  75.00%.  Furthermore,  the  coding  schema  was 
created  through  latent  identification  of  the  key  issues  causing  reliability  coefficients  to  be 
expected  to  receive  lower  reliability  scores  (Neuendorf,  2002).  Therefore,  the  remaining 
13  issues’  reliability  coefficients  were  accepted  as  reliable  and  the  issues  were  retained  in 
the  measurement  instrument. 

With  both  the  reproducibility  and  stability  of  the  instrument  established  the 
internal  validity  of  the  instrument  was  confirmed.  The  validated  coding  schema  was  then 
utilized  to  answer  the  final  three  investigative  questions. 

Overview  of  Findings 

This  content  analysis  initially  documented  35  latent  issues  across  the  52  articles 
identified  during  the  sampling  procedure.  One  article  was  selected  to  be  reviewed  by  all 
the  coders  to  provide  a  measurement  of  inter-rater  reliability.  From  this  assessment  12 
issues  were  identified  as  unreliable.  The  remaining  issues  answer  the  third  investigative 
question  of  this  research: 

3.  What  does  the  literature  identify  as  the  issues  that  must  be  addressed  to  effectively 
manage  an  enterprise  architecture? 

The  23  issues  are  presented  in  Table  5.  The  respective  reliability  coefficients  are 
also  reported  to  ensure  low  reliabilities  were  not  obscured.  The  top  ten  issues  are 
reported  as  highly  reliable.  The  remaining  13  issues  must  also  be  addressed,  but  as  can 
be  seen  their  respective  reliability  rating  is  below  70%.  Such  a  low  reliability  rating 
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causes  the  assessment  to  become  difficult  to  interpret  (Neuendorf,  2002).  Nevertheless, 
this  research  was  exploratory  in  nature  making  it  difficult  to  maintain  objectivity  during 
the  process  of  issue  identification.  Therefore,  these  issues  are  still  reported. 


Table  5.  Reliability  Coefficient  for  Each  Issue 


Issue 

Reliability 

Coefficient 

Understanding  the  business  processes  allows  the  architecture  to  ensure  the  implementation  of  IT  systems  that 
will  match  the  required  business  needs 

0.9167 

The  enterprise  architecture  must  have  senior  management  support 

0.9167 

Architecture  must  be  capable  of  adapting  or  modifying  itself  to  reflect  changes  in  strategic  objectives, 
reorganization  and/or  business  process  changes 

0.8750 

Identify  gaps  between  baseline  and  established  targets 

0.8750 

Gain  knowledgeable  architecture  resources  from  consultants 

0.8750 

Evolve  the  architecture  over  time  in  a  iterative  step  by  step  transition  plan  and  analyze  how  changes  in  the 
organization's  mission,  functions,  and  needs  might  have  an  effect  on  system  development 

0.8333 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear,  meaningful,  and  quantifiable 

0.8333 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data  integration  and  data  sharing  across 
diverse  applications 

0.8333 

A  culture  must  be  developed  that  focuses  on  the  importance  of  coordinated  planning  between  business  and  IT 

0.8333 

Architecture  development  must  be  flexible  to  accommodate  a  range  of  architectures  and  functional  areas 

0.7500 

Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

0.6667 

Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable,  measurable,  and  reduces 

0.6667 

Feedback  is  received  on  performance  so  future  architecture  changes  will  be  more  successful 

0.6250 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  individual  business  units--best 
practice  processes  can  be  recognized  and  implemented  across  the  entire  organization 

0.6250 

Framework  guides  architecture  design  and  investment  decision  making 

0.5833 

Start  with  doable  and  critical  system  development  projects 

0.5833 

Common  understanding  and  conformance  to  architecture  principles  and  standards  leads  to  consistent 
enforcement  of  guidance,  informed  system  development  decisions,  and  reduced  redundancy 

0.5833 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the  data  that  is  provided 

0.5833 

Development  of  an  architecture  must  include  the  business/functional  users 

0.5833 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users  with  the  ability  and 
authority  to  answer  human,  technical,  and  business  questions  and  carry  out  assigned  responsibilities 

0.5833 

Define  the  target  business  view 

0.5417 

Determine  target  architecture  (Where  we  want  to  be) 

0.5417 

An  architecture  is  a  tool  that  allows  the  organization  to  gain  a  competitive  by  being  a  tool  that  can  assist  in 
making  the  decision  whether  or  not  to  implement  new  technologies  and/or  retain  legacy  systems 

0.5417 

The  fourth  investigative  question  was  then  answered  by  using  the  primary 
researcher’s  validated  codebook.  A  tally  was  added  for  each  issue  to  find  out  how  many 
times  each  issue  appears  across  all  52  articles.  Table  6  presents  the  findings  that  answer 
the  following  question: 
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4.  Which  issues  have  the  most  relevance? 


Table  6.  Issue  Relevance 


Issue 

Count 

Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

33 

Architecture  must  be  capable  of  adapting  or  modifying  itself  to  reflect  changes  in  strategic 
objectives,  reorganization  and/or  business  process  changes 

27 

Evolve  the  architecture  over  time  in  a  iterative  step  by  step  transition  plan  and  analyze  how 
changes  in  the  organization's  mission,  functions,  and  needs  might  effect  system  development 

27 

The  enterprise  architecture  must  have  senior  management  support 

27 

Development  of  an  architecture  must  include  the  business/functional  users 

26 

Understanding  the  business  processes  allows  the  architecture  to  ensure  the  implementation  of 

IT  systems  that  will  match  the  required  business  needs 

20 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data  integration  and 
data  sharing  across  diverse  applications 

19 

Determine  target  architecture  (Where  we  want  to  be) 

16 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users  with 
ability  and  authority  to  answer  human,  technical,  and  business  questions  and  carry  out  assigned 
tasks 

15 

Framework  guides  architecture  design  and  investment  decision  making 

14 

Common  understanding  and  conformance  to  architecture  principles  and  standards  leads  to 
enforcement  of  guidance,  informed  system  development  decisions,  and  reduced  redundancy 

14 

Define  the  target  business  view 

13 

An  architecture  allows  the  organization  to  gain  a  competitive  by  being  a  tool  that  can  assist  in 
making  the  decision  whether  or  not  to  implement  new  technologies  and/or  retain  legacy 
systems 

13 

Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable, 
measurable,  and  reduces  redundancy 

11 

Identify  gaps  between  baseline  and  established  targets 

9 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear,  meaningful, 
and  quantifiable 

9 

Gain  knowledgeable  architecture  resources  from  consultants 

8 

Architecture  development  must  be  flexible  to  accommodate  a  range  of  architectures  and 
functional  areas  requirements 

7 

Start  with  doable  and  critical  system  development  projects 

7 

Feedback  is  received  on  performance  so  future  architecture  changes  will  be  more  successful 

7 

The  culture  must  focus  on  importance  of  coordinated  planning  between  business  and  IT 

6 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  individual  business 
units— best  practice  processes  can  be  recognized  and  implemented  across  the  entire  organization 

D 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the  provided 
data 

The  category  each  issue  belonged  to  on  the  codebook  used  during  the  content 
analysis  was  introduced  to  answer  the  fifth  investigative  question: 

5.  What  does  the  literature  identify  as  the  underlying  factors  driving  these  issues? 
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Table  7  answers  this  question  by  reporting  the  respective  category  of  each  issue. 


Table  7.  Underlying  Factors 


Factor 

Issue 

Operational 

Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

Flexibility 

Architecture  must  be  capable  of  adapting  or  modifying  itself  to  reflect  changes  in 
strategic  objectives,  reorganization  and/or  business  process  changes 

Maintenance 

Evolve  the  architecture  over  time  in  a  iterative  transition  plan  and  analyze  how 
changes  in  the  organization's  mission,  functions,  and  needs  might  effect  system 
development 

Organizational 

The  enterprise  architecture  must  have  senior  management  support 

Organizational 

Development  of  an  architecture  must  include  the  business/ functional  users 

Processes 

Understanding  the  business  processes  allows  the  architecture  to  ensure  the 
implementation  of  IT  systems  that  will  match  the  required  business  needs 

Data 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data 
integration  and  data  sharing  across  diverse  applications 

Development 

Determine  target  architecture  (Where  we  want  to  be) 

Organizational 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users 
with  the  ability  and  authority  to  answer  human,  technical,  and  business  questions  and 
carry  out  assigned  responsibilities 

Development 

Framework  guides  architecture  design  and  investment  decision  making 

Maintenance 

Common  understanding  and  conformance  to  architecture  principles  leads  to  guidance 
enforcement,  informed  system  development  decisions,  and  reduced  redundancy 

Development 

Define  the  target  business  view 

Technology 

An  architecture  allows  the  organization  to  gain  a  competitive  by  assisting  in  making 
the  decision  whether  or  not  to  implement  new  technologies  and/or  retain  legacy 
systems 

Processes 

Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable, 
measurable,  and  reduces  redundancy 

Development 

Identify  gaps  between  baseline  and  established  targets 

Maintenance 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear, 
meaningful,  and  quantifiable 

Organizational 

Gain  knowledgeable  architecture  resources  from  consultants 

Flexibility 

Architecture  development  must  be  flexible  to  accommodate  a  range  of  architectures 
and  functional  areas  requirements 

Maintenance 

Feedback  is  received  on  performance  to  make  future  architecture  changes  successful 

Development 

Start  with  doable  and  critical  system  development  projects 

Organizational 

A  culture  must  be  developed  that  focuses  on  the  importance  of  coordinated  planning 
between  business  and  IT 

Processes 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  business 
units— best  practice  processes  are  recognized  and  implemented  across  the  organization 

Data 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the 
data 

The  tables  presented  in  this  section  provide  a  snapshot  of  the  content  analysis  and 


are  used  to  answer  the  three  remaining  investigative  questions.  To  reach  an 


63 


understanding  of  which  issues  the  Air  Force  must  address  to  effectively  manage  its 
enterprise  architecture  a  more  in-depth  analysis  must  be  attempted.  The  next  section 
makes  this  attempt  in  an  effort  to  reach  the  goal  of  this  study. 

Discussion 

To  implement  and  manage  the  Air  Force  Enterprise  Architecture  the  Air  Force 
Communications  Agency  (AFCA)  transitioned  their  SCOPE  Network  teams  that  focused 
on  optimizing  and  securing  the  base  networks  to  SCOPE  EDGE  teams.  SCOPE  Net’s 
mission  was  to  optimize  existing  base  networks.  Today,  SCOPE  EDGE  changed  their 
view  of  the  Air  Force  network  from  a  collection  of  individual  bases  to  a  network 
enterprise  (Hoeft,  2004). 

This  research  synthesizes  the  literary  efforts  of  both  academic  and  professional 
authors  to  provide  AFCA’s  SCOPE  EDGE  mission  with  a  strategic  guidepost  that 
identifies  and  analyzes  the  key  issues  affecting  the  Air  Force’s  ability  to  manage  its 
enterprise  architecture.  The  previous  section  identified  the  key  issues,  but  as  Table  5 
shows  there  is  no  consistency  in  the  factors  which  cause  these  issues  to  be  relevant.  For 
example,  the  five  most  relevant  issues  are  driven  by  four  clearly  distinct  factors.  Even 
when  the  top  ten  most  relevant  issues  are  reviewed  seven  different  underlying  factors  are 
identified  as  the  reason  for  their  relative  importance.  Therefore,  the  conclusion  reached 
from  this  research  is  that  to  effectively  manage  an  enterprise  architecture  the  Air  Force 
must  not  focus  on  one  organizational  factor.  Instead,  to  effectively  manage  the  enterprise 
architecture  a  holistic  approach  must  be  taken. 
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From  the  analysis,  two  interesting  findings  are  worth  mentioning.  The  first  is  that 
of  the  top  ten  most  relevant  issues  three  of  them  are  driven  by  organizational  factors.  The 
researcher  is  hesitant  to  identify  this  as  a  clear  answer  to  the  goal  of  this  research. 
However,  it  can  be  stated  that  the  effective  management  of  the  enterprise  architecture 
requires  the  dedicated  support  of  the  entire  organization.  The  other  finding  was  that 
technology  was  only  found  as  the  underlying  factor  for  one  of  the  issues.  Therefore,  it 
can  be  said  that  approaching  the  management  of  the  enterprise  architecture  from  a 
technological  standpoint  may  lead  to  failure. 

Applying  these  findings  to  AFCA’s  responsibility  to  manage  the  Air  Force’s 
Enterprise  Architecture  reveals  a  gap  between  the  current  enterprise  management 
techniques  and  the  researcher’s  findings.  These  findings  suggest  SCOPE  EDGE  should 
consider  expanding  its  technological  perspective  towards  a  holistic  management 
approach  to  accomplish  the  new  mission  of  strategic  network  advocacy  and  enterprise 
level  assessment.  The  remainder  of  this  section  discusses  the  key  issues  and  their 
underlying  factors  as  identified  by  the  researcher.  This  analysis  of  the  findings  provides 
AFCA  with  a  guidepost  for  their  efforts  to  manage  the  Air  Force’s  Enterprise 
Architecture. 

As  stated  above,  there  was  no  clear  and  concise  answer  to  how  an  enterprise 
architecture  should  be  managed.  For  this  reason,  the  primary  researcher  identified 
common  themes  across  the  identified  issues.  This  allowed  the  researcher  to  consolidate 
the  findings  into  a  manageable  set  of  issues  and  then  discuss  the  possible  implications. 
The  first  issue  is  the  only  one  that  stands  out  above  the  other  issues.  It  was 
acknowledged  in  over  67%  of  the  article  analyzed.  This  issue  states  the  enterprise 
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architecture  must  be  tied  directly  to  the  organization’s  operational  mission  and  vision. 
Furthermore,  four  other  issues  appeared  in  over  50%  of  all  the  articles  analyzed.  Table  8 
reports  the  top  five  issues  identified  by  the  primary  researcher.  As  can  be  seen,  these  five 
issues  have  an  underlying  theme  appearing  in  each  one.  In  short,  each  issue  has  a  direct 
impact  on  the  organization’s  operational  mission  and  vision. 


Table  8.  Top  Five  most  Relevant  Issues 


Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

33 

Architecture  must  be  capable  of  adapting  or  modifying  itself  to  reflect  changes  in  strategic 
objectives,  reorganization  and/or  business  process  changes 

27 

Evolve  the  architecture  over  time  in  a  iterative  step  by  step  transition  plan  and  analyze  how 
changes  in  the  organization's  mission,  functions,  and  needs  might  have  an  effect  on  system 
development 

27 

The  enterprise  architecture  must  have  senior  management  support 

27 

Development  of  an  architecture  must  include  the  business/functional  users 

26 

The  number  of  times  these  five  issues  appeared  as  compared  to  the  remaining  1 8 
issues  expounds  upon  the  importance  of  tying  the  enterprise  architecture  directly  to  the 
mission  and  vision.  Overall,  the  35  issues  included  in  the  coding  schema  resulted  in  a 
total  of  304  issues  being  identified  throughout  the  52  articles.  These  five  issues  account 
for  over  46%  of  this  cumulative  total.  It  must  be  noted  the  first  three  issues  all  directly 
make  reference  to  how  the  enterprise  architecture  must  be  connected  to  the  mission.  The 
remaining  two  issues  do  not  make  this  direct  connection.  In  spite  of  this,  it  is  not  difficult 
to  understand  that  the  senior  manager  sets  the  mission  and  vision  while  the  business  and 
functional  users  ensure  the  mission  is  accomplished.  In  the  end,  the  importance  of  tying 
the  enterprise  architecture  to  the  organization  is  stressed  throughout  the  analyzed 
literature. 

Individually  analyzing  the  next  nine  issues  causes  them  to  appear  to  be  unrelated. 
As  the  issues  are  bridged  together,  the  common  theme  of  controlling  the  enterprise 
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architecture  emerges.  Just  as  an  enterprise  architecture  has  several  views  that  must  be 
integrated  together,  controlling  an  enterprise  architecture  must  be  approached  from 
several  angles.  Six  different  views  of  control  were  documented  during  the  content 
analysis.  The  views  and  their  related  issues  are  put  forward  in  Table  9.  These  nine  issues 
must  be  addressed  to  ensure  the  enterprise  architecture  is  properly  controlled. 


Table  9.  Control  Issues 


Data 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data  integration 
and  data  sharing  across  diverse  applications 

Development 

1.  Framework  guides  architecture  design  and  investment  decision  making 

2.  Determine  target  architecture  (Where  we  want  to  be) 

3.  Define  the  target  business  view 

Maintenance 

Common  understanding  and  conformance  to  architecture  principles  leads  to  consistent 
guidance  enforcement,  informed  system  development  decisions,  and  reduced  redundancy 

Organizational 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users 
with  the  ability  and  authority  to  answer  human,  technical,  and  business  questions  and 
carry  out  assigned  responsibilities 

Processes 

1.  Understanding  the  business  processes  allows  the  architecture  to  ensure  the 
implementation  of  IT  systems  that  will  match  the  required  business  needs 

2.  Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable, 
measurable,  and  reduces  redundancy 

Technology 

An  architecture  is  a  tool  that  allows  the  organization  to  gain  a  competitive  by  being  a  tool 
that  can  assist  in  making  the  decision  whether  or  not  to  implement  new  technologies 
and/or  retain  legacy  systems 

The  ability  to  exploit  an  enterprise  architecture  to  direct,  measure,  and  capture 
change  accounts  for  just  over  15%  of  total  number  of  issues  identified  in  all  the  articles. 
This  theme  is  seen  in  the  next  six  issues  which  are  presented  in  the  following  table: 

Table  10.  Direct,  Manage,  and  Capture  Change  Issues 

Identify  gaps  between  baseline  and  established  targets 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear,  meaningful, 

and  quantifiable _ 

Gain  knowledgeable  architecture  resources  from  consultants 

Architecture  development  must  be  flexible  to  accommodate  a  range  of  architectures  and 
functional  areas  requirements 

Feedback  is  received  on  performance  so  future  architecture  changes  will  be  more  successful 
Start  with  doable  and  critical  system  development  projects 
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The  low  frequency  score  of  these  issues  demonstrates  the  lack  of  attention  placed 
on  them.  This  adds  support  to  the  motivation  for  completing  this  research.  The 
development  and  implementation  of  enterprise  architectures  has  been  discussed  across  a 
wide  variety  of  literature.  Conversely,  the  topic  of  managing  an  enterprise  architecture 
has  not  been  adequately  addressed.  Each  of  these  issues  covers  a  different  aspect  of 
managing  an  enterprise  architecture.  As  can  be  seen,  additional  emphasis  must  be  placed 
on  properly  managing  an  enterprise  architecture. 

The  final  three  issues  are  presented  below  in  Table  11.  These  issues  all  are 
concerned  with  centralized  coordination  of  the  enterprise  architecture. 

Table  11.  Central  Coordination  Issues 

A  culture  must  be  developed  that  focuses  on  the  importance  of  coordinated  planning  between 
business  and  IT 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  individual  business 
units— best  practice  processes  can  be  recognized  and  implemented  across  the  entire 
organization 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the  data  that 
is  provided 


Once  again  these  three  issues  received  the  least  amount  of  attention  throughout  the 
articles  selected  for  the  content  analysis.  This  finding  can  also  be  attested  to  the  fact  that 
there  is  a  void  in  the  body  of  knowledge  concerning  the  management  of  an  enterprise 
architecture. 

The  23  issues  were  reviewed  causing  four  themes  to  be  identified:  (1)  tying  the 
enterprise  architecture  to  the  operational  mission,  (2)  controlling  the  enterprise 
architecture,  (3)  directing,  measuring,  and  capturing  change,  and  (4)  centralized 
coordination.  These  four  themes  provide  AFCA  with  the  necessary  foundation  required 
to  begin  to  effectively  manage  the  Air  Force  Enterprise  Architecture.  In  addition,  the 
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issues  underlying  each  theme  offer  enterprise  architecture  practitioners  with  a  means  to 
develop  relevant  operational  measurements  to  determine  how  mature  the  enterprise 
architecture  is  in  relation  to  set  standards. 

Limitations  of  Results 

During  the  computation  and  analysis  of  the  results  four  additional  limiting  factors 
were  identified.  These  factors  include  potential  confounding  factors  affecting  the 
stability  measurement,  the  normality  of  the  distributions,  the  low  reproducibility 
measurement,  and  the  inability  to  generalize  about  the  findings.  This  section  discusses 
each  of  these  limiting  factors. 

The  independent  t-test  proved  the  coding  schema  was  stable  across  two  groups 
measuring  identical  articles.  This  test  requires  the  samples  to  be  independently  and 
randomly  selected  from  the  population.  In  this  study  the  two  groups  were  not  selected  in 
this  manner.  Instead  the  groups  were  chosen  by  stratifying  the  population  of  coders  by 
operational  and  educational  experience.  Another  potential  confounding  factors  is  that  all 
the  coders  were  enrolled  in  an  Enterprise  Architecture  course  while  the  study  was 
conducted.  These  two  confounders  factors  may  have  affected  the  results  of  the  test  for 
stability. 

To  establish  the  internal  validity  and  reliability  of  the  coding  schema  the 
frequency  distribution  of  each  percent  agreement  measurement  was  analyzed.  As 
reported  in  the  pilot  and  full  study,  the  assumption  of  normality  could  not  be  supported 
for  each  distribution.  To  overcome  this  limitation  the  sample  size  would  have  to  be 
increased  to  allow  the  measurements  to  approach  a  normal  distribution. 
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Another  limiting  factor  was  the  original  coding  schema’s  low  percent  agreement 
for  the  inter-rater  reliability  measurement.  Once  the  unreliable  issues  were  removed  from 
the  schema  the  mean  increased  from  59.40%  to  70.65%.  This  demonstrates 
improvements  could  have  been  made  by  clarifying  terms  used  in  the  coding  schema. 

This  would  decrease  the  variability  caused  by  the  subjective  judgment  of  the  coders  and 
allowed  for  an  effective  analysis  of  the  articles  included  in  the  study. 

The  data  collection  process  used  was  nonrandom,  limiting  the  ability  to  make 
generalizations  of  the  findings  to  the  entire  population  of  enterprise  architecture 
literature.  The  articles  chosen  for  this  study  were  initially  selected  from  the  top  ten 
management  journals  to  remove  the  researcher’s  subjective  bias  during  the  article 
selection.  In  ensuring  that  bias  was  removed,  the  article  collection  was  constrained  to 
only  these  journals.  The  imposed  constraints  caused  the  sample  to  become  nonrandom. 

Furthermore,  the  identified  articles’  keywords  were  then  used  to  perform  an 
online  search  for  data  sources.  Online  searches  are  plagued  by  the  difficulty  to  establish 
a  population  and  a  sampling  frame  (Neuendorf,  2002).  Exhaustive  searches  were 
performed  to  include  conference  proceedings,  a  variety  of  peer-review  articles, 
professional  papers,  and  government  reports.  However,  without  a  defined  population  a 
truly  random  sample  could  not  be  selected. 

Summary 

The  results  of  this  research  were  presented.  First,  an  agreed  upon  operational 
definition  was  presented  to  answer  the  first  two  investigative  questions.  Once  the 
contextual  basis  had  been  established,  the  descriptive  statistics  of  the  sample  set  was 
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reported.  Next,  the  results  from  the  steps  taken  to  validate  the  coding  schema  were 
described.  The  full  study  was  then  conducted.  From  this  study  the  reliability  of  the 
coding  schema  was  established  by  analyzing  the  reproducibility  and  stability  of  the 
measurement  instrument.  With  the  validity  and  reliability  of  the  measurement  instrument 
established,  it  was  then  utilized  to  examine  the  primary  researcher’s  coding  results  to 
answer  the  final  three  investigative  questions  of  this  research.  This  analysis  was  then 
used  to  discuss  the  interpretation  of  the  results.  Finally,  the  limitations  of  the  study’s 
results  were  presented. 


71 


V.  Conclusion 


Closing  Remarks 

The  results  of  this  research  has  implications  not  only  for  AFCA.  The  Air  Force  is 
transitioning  from  an  acquisition  system  based  upon  platform  specific  purchases  to  a 
capabilities  based  force.  Ensure  an  organization’s  data,  information,  personnel,  and 
information  systems  are  being  utilized  to  achieve  the  identified  capability  requirements 
can  be  accomplished  by  employing  an  enterprise  architecture.  The  following  two 
sections  discuss  the  implications  for  the  sponsor  of  this  research,  AFCA  and  the  possible 
impact  a  properly  manage  enterprise  architecture  may  have  on  the  Air  Force. 

Implications  for  AFCA 

To  comply  with  Section  5 125  of  the  Clinger-Cohen  Act  and  OMB  circular  A-130 
the  Air  Force  has  developed  an  enterprise  architecture  to  guide  its  investment  in 
information  systems.  The  responsibility  of  implementing  and  managing  this  enterprise 
architecture  has  become  the  responsibility  of  the  Air  Force  Communications  Agency.  In 
response,  AFCA  transitioned  from  SCOPE  Network  teams  that  focused  on  optimizing 
and  securing  the  base  networks  to  what  are  called  SCOPE  EDGE  teams.  While  SCOPE 
Net’s  mission  was  network  optimization,  SCOPE  EDGE  has  increased  the  level  of 
responsibility  to  include  strategic  network  advocacy  and  enterprise  level  assessment 
(Hoeft,  2004). 

AFCA’s  new  mission  has  caused  SCOPE  EDGE  to  no  longer  view  each  base 
network  individually.  Instead,  the  focus  has  shifted  to  ensure  compliance  with 
architecture  standards  for  the  entire  network  enterprise.  Unfortunately,  this  focus  only 
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accounts  for  the  technological  view  of  the  Air  Force’s  Enterprise  Architecture.  This 
focus  places  an  emphasis  on  the  hardware  and  software  that  comprises  the  enterprise 
architecture,  but  neglects  the  other  architecture  views. 

In  response  to  SCOPE  EDGE’s  network-based  strategic  perspective,  this  research 
was  conducted  to  synthesize  the  literary  efforts  of  both  academic  and  professional  authors 
to  provide  AFCA’s  SCOPE  EDGE  mission  with  a  strategic  guidepost  that  identifies  and 
analyzes  the  key  issues  affecting  the  Air  Force’s  ability  to  manage  its  enterprise 
architecture.  Chapter  4’s  presentation  of  the  findings  and  discussion  achieved  the  stated 
objective  of  this  research.  However,  there  was  no  clear  and  concise  answer  to  the  central 
research  question.  Ultimately,  the  researcher  concluded  that  the  effective  management  of 
an  enterprise  architecture  requires  AFC  A  to  not  focus  on  one  factor;  instead,  a  holistic 
management  approach  must  be  taken. 

Currently,  SCOPE  EDGE  is  measuring  the  Network  Health  of  the  network 
enterprise  through  the  use  of  a  Network  Maturity  Model.  However,  AFCA’s 
responsibility  is  to  provide  enterprise  level  assessment.  Since  the  focus  has  expanded 
from  the  network  to  the  entire  enterprise  architecture,  SCOPE  EDGE  can  no  longer 
afford  to  focus  on  just  the  technology  that  comprises  the  enterprise.  The  research 
findings  show  AFCA’s  strategic  perspective  should  be  broadened  to  manage  all  aspects 
of  the  Air  Force’s  Enterprise  Architecture. 

To  provide  the  Air  Force  Communications  Agency  with  a  foundation  to  begin  to 
manage  the  Air  Force’s  Enterprise  Architecture  the  researcher  identified  common  themes 
across  the  identified  issues.  The  23  issues  were  reviewed  causing  four  themes  to  be 
identified:  (1)  tie  the  enterprise  architecture  to  the  Air  Force’s  operational  mission,  (2) 
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control  the  enterprise  architecture,  (3)  direct,  measure,  and  capture  change,  and  (4) 
centralize  coordination.  From  these  four  themes  an  enterprise  architecture  maturity 
model  could  be  developed  to  replace  the  current  Network  Maturity  Model.  This  would 
allow  SCOPE  EDGE  to  broaden  their  perspective  from  focusing  on  only  the 
technological  components  of  the  enterprise  to  having  the  ability  to  effectively  manage  the 
entire  enterprise  architecture. 

To  develop  an  enterprise  architecture  maturity  model  the  issues  underlying  the 
identified  themes  provide  AFC  A  with  a  means  to  develop  relevant  statistical 
measurements  to  determine  how  mature  the  enterprise  architecture  is  in  relation  to  set 
standards.  In  turn,  the  ability  to  measure  the  salient  issues  when  managing  the  enterprise 
architecture  permits  managers  to  assess  progress  toward  the  desired  end  and  to  take 
corrective  action  to  address  unacceptable  deviations.  In  the  end,  SCOPE  EDGE  will 
have  an  assessment  tool  to  determine  if  the  Air  Force  can  effectively  manage  its 
enterprise  architecture. 

Implications  for  the  Air  Force 

The  Air  Force’s  currently  defines  its  enterprise  architecture  as  three  inter-related 
components  consisting  of  a  system  view,  operational  view,  and  technical  view.  However, 
these  three  components  do  not  incorporate  the  people,  data,  or  motivation  views  as 
proposed  by  Zachman’s  enterprise  architecture  framework  (Zachman  1987).  The 
absence  of  these  views  hinders  the  Air  Force’s  ability  to  utilize  its  enterprise  architecture 
to  tie  its  information  systems  directly  to  the  operational  mission. 

As  the  Air  Force  transitions  to  a  capabilities  based  force,  the  motivation  view, 
which  is  absent  from  the  Air  Force’s  framework,  would  allow  the  architect  to  detennine 
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how  an  infonnation  system  could  possibly  enable  the  required  capability.  Today 
technological  solutions  are  being  developed  prior  to  understanding  the  underlying 
motivations  for  the  identified  requirement.  Instead,  the  motivation  view  must  first  be 
defined  to  determine  if  a  technological  solution  is  feasible  and/or  appropriate.  In  turn, 
this  would  ensure  the  information  systems  are  directly  tied  to  the  operational  mission. 

In  the  Air  Force  of  today,  there  exists  an  ever  widening  gap  between  the 
information  system  user  and  the  provider  of  that  same  system.  The  lack  of  understanding 
of  how  the  system  is  employed  in  the  operational  environment  causes  technological 
decisions  to  be  made  that  could  potentially  cause  a  degradation  of  the  services  provided 
by  the  system.  By  expanding  the  Air  Force’s  current  enterprise  architecture  to  include  a 
people  view  a  bridge  would  be  placed  across  this  current  gap  that  exists  between  the 
operator  and  the  user. 

The  Air  Force  has  placed  its  focus  on  developing  its  technical  view  of  its 
enterprise  architecture;  however,  this  view  only  pertains  to  the  enterprise  infrastructure. 
Equally  important  is  both  the  business  architecture  and  infonnation  architecture  views. 
The  Air  Force’s  enterprise  architecture  does  address  the  business  architecture  in  its 
operational  component,  but  once  again  the  infonnation  architecture  has  no  central 
coordinating  mechanism  in  place. 

As  can  be  seen,  efforts  must  be  made  by  the  Air  Force  to  expand  its  enterprise 
architecture  to  integrate  all  views  of  the  enterprise  architecture.  This  holistic  approach 
not  only  will  confinn  it  is  properly  managed,  but  it  will  also  ensure  the  information 
system  is  directly  supporting  the  organization’s  operational  mission. 
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Conclusion 


This  research  contributes  to  both  practitioners  and  academic  researchers.  As  the 
Air  Force  continues  to  implement  and  manage  its  enterprise  architecture  this  study 
provides  practicing  enterprise  architects  with  a  consolidated  list  of  the  key  issues  that 
should  be  addressed  to  manage  an  enterprise  architecture.  In  addition,  it  demonstrates 
that  no  one  factor  leads  to  the  successful  management  of  the  enterprise.  Instead,  the 
architect  must  focus  on  all  aspects  of  managing  the  architecture.  At  the  same  time,  this 
research  fulfills  an  academic  void.  To  date  the  current  stream  of  research  has  not 
identified  the  key  issues  involved  in  the  successful  management  of  an  enterprise 
architecture  or  the  underlying  factors  driving  these  issues.  This  study  laid  the 
groundwork  to  remove  this  obstacle. 

Limitations  of  Research 

Limiting  factors  emerged  during  this  research.  These  limitations  were  identified 
in  the  validity,  reliability,  and  generalizability  of  the  research  and  are  addressed  in  the 
following  three  sections. 

Reliability 

The  measurement  instrument’s  reproducibility  was  reported  by  measuring  each 
issue’s  percent  agreement  between  the  primary  researcher  and  the  24  coders.  However, 
Cohen’s  Kappa  is  a  stronger  measurement  coefficient  because  it  removes  chance 
agreement  (Neuendorf,  2002).  Since  this  research  had  35  issues,  chance  agreement,  even 
though  possible,  was  not  seen  as  a  confounding  factor.  In  addition,  the  original  coding 
schema’s  low  percent  agreement  was  another  limiting  factor.  Once  the  unreliable  issues 
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were  removed  from  the  schema  the  mean  percent  agreement  increased  from  59.40%  to 
70.65%.  Improvements  could  have  been  made  by  clarifying  terms  used  in  the  coding 
schema.  This  would  decrease  the  variability  caused  by  the  subjective  judgment  of  the 
coders  and  allow  for  an  effective  analysis  of  the  articles  included  in  the  study. 

The  coding  was  performed  by  12  groups  each  consisting  of  two  students.  To 
establish  the  instruments  stability  the  two  students  were  each  assigned  the  same  articles. 
However,  these  students  were  all  enrolled  in  an  Enterprise  Architecture  class  during  data 
collection.  An  independent  group  of  coders  was  not  identified  to  ensure  the  educational 
experience  was  not  a  confounding  factor  on  the  results  obtained.  Therefore,  even  though 
it  was  addressed,  stability  was  identified  as  a  limitation  to  this  research  effort. 

Validity 

In  this  research,  both  the  external  and  internal  validity  had  shortcomings.  The 
internal  validity  had  to  be  addressed  because  each  of  the  issues  included  in  the  codebook 
were  not  identified  before  the  content  analysis.  This  did  not  allow  the  match  up  of  a 
conceptual  definition  and  an  operational  measurement  (Neuendorf,  2002).  To  overcome 
this  limitation  an  impartial  process  of  issue  development  was  used.  Four  co-researchers 
were  each  assigned  a  subset  of  the  articles  included  in  the  content  analysis  to  address  the 
potential  for  personal  bias  and  to  prevent  errors  of  judgment  and  misinterpretation  of  the 
text  by  the  primary  researcher. 

The  external  validity  is  substantiated  from  the  ability  of  others  to  repeat  the  study 
with  a  different  set  of  messages  (Neuendorf,  2002).  Since  there  were  no  prior  research 
efforts  identifying  how  an  enterprise  architecture  should  be  managed,  this  research 
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employed  an  exploratory  strategy.  Therefore,  there  are  no  successful  replications  to 
support  the  external  validity. 

Generalizability 

The  data  collection  process  used  was  nonrandom,  limiting  the  ability  to  make 
generalizations  of  the  findings  to  the  entire  population  of  enterprise  architecture 
literature.  The  articles  chosen  for  this  study  were  initially  selected  from  the  top  ten 
information  system  management  journals  to  remove  the  researcher’s  subjective  bias 
during  the  article  selection.  In  ensuring  that  bias  was  removed,  the  article  collection  was 
constrained  to  only  these  journals.  The  imposed  constraints  caused  the  sample  to  become 
nonrandom.  Furthermore,  the  articles’  keywords  were  used  to  perform  an  online  search 
for  data  sources.  Online  searches  are  plagued  by  the  difficulty  to  establish  a  population 
and  a  sampling  frame  (Neuendorf,  2002).  Without  a  defined  population  a  truly  random 
sample  could  not  be  selected. 

Concurrent  Thesis  Research  Efforts 

It  is  important  to  note  that  there  has  been  concurrent  research  performed  for 
AFC  A,  the  sponsor  of  this  research.  First  Lieutenant  Charlie  Boyd’s  focus  his  efforts  on 
the  Air  Force’s  enterprise  infrastructure  by  examining  Air  Force  installation-level 
networks  that  contribute  to  DoD’s  interoperability  and  integration.  He  focused  on 
installation-level  wide  area  and  local  area  networks,  WANs  and  LANs  respectively, 
which  represent  the  lowest  Air  Force  portion  of  the  Global  Information  Grid.  Currently, 
services  cannot  support  the  DoD  blueprint  for  development  and  transformation  without 
addressing  installation-level  networks.  Therefore,  his  research  study  explores  Air  Force 
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installation-level  or  base  area  network  (BAN)  architectures.  Lt  Boyd’s  thesis  was 
completed  over  the  same  time  period  as  this  thesis  was  undertaken  and  is  scheduled  to  be 
completed  and  published  in  March  2005. 

Suggestions  for  Further  Study 

Qualitative  research  can  never  capture  objective  reality;  therefore,  further  studies 
are  required  to  secure  an  in-depth  understanding  of  the  issues  the  Air  Force  must  address 
to  effectively  manage  its  enterprise  architecture.  The  results  of  this  study  establish  a 
foundation  that  can  be  used  as  a  stepping  stone  for  a  multitude  of  follow-on  studies. 

A  follow-on  study  duplicating  this  methodology,  but  using  a  different  coding 
process  is  recommended.  One  proposal  is  to  add  weights  to  the  presence  of  an  issue 
within  each  article.  This  would  separate  issues  that  are  mentioned  in  the  article,  but  not 
well  developed  from  issues  that  are  the  driving  reason  the  article  is  written.  The  effect  of 
this  coding  process  would  be  to  address  the  level  of  importance  of  the  identified  issues 
across  all  the  articles  included  in  the  content  analysis.  In  addition,  the  follow-on  study 
should  identify  a  different  collection  of  articles  by  reviewing  additional  academic 
journals  and  using  another  set  of  key  word  searches  or  utilizing  other  database  search 
engines.  Increasing  the  sample  size  and  then  comparing  the  results  of  the  new  articles  to 
the  original  set  of  articles  would  allow  the  results  to  be  generalized  to  the  entire 
population  of  enterprise  architecture  literature. 

With  these  results,  the  question  of  what  are  the  key  issues  the  Air  Force  must 
address  would  be  answered.  The  next  question  to  answer  is  how  are  these  key  issues 
being  addressed?  A  multi-site  case  study  focusing  on  the  management  practice  of 
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enterprise  architects  within  each  Air  Force  MAJCOM,  Direct  Reporting  Unit,  and  Field 
Operating  Agency  would  expand  upon  the  identified  issues.  Analyzing  these  particular 
organizations  would  provide  a  greater  understanding  of  the  issues  and  their  underlying 
factors.  The  ability  to  find  architects  who  are  managing  all  aspects  of  the  enterprise 
architecture  may  prove  to  be  challenging,  but  this  study  would  provide  the  Air  Force  with 
an  independent  assessment  of  their  current  level  of  architecture  maturity. 

Finally,  each  of  the  identified  issues  could  be  used  to  form  an  interview  and 
survey.  These  measurement  instruments  could  then  be  used  to  conduct  a  Pre-Interview 
Survey  of  Air  Force  enterprise  architects.  The  survey  would  identify  potential  architect 
experts  and  detennine  how  pertinent  the  identified  issues  are  to  practitioners.  In  addition, 
the  interview  could  determine  how  the  identified  issues  can  be  or  have  been 
operationalized  into  measurable  statistics.  This  would  provide  a  means  to  measure  where 
on  the  enterprise  architecture  capability  maturity  model  an  organization  is  located. 
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Appendix  B:  Original  Codebook 


Article  Number 

Key  Issues 

Flexibility 

Architecture  development  must  be  flexible  to  accommodate  range  of  architectures  and  functional  areas 
requirements 

Architecture  must  be  adaptable/modifiable  to  allow  for  reviewing  and  updating  the  architecture  to  reflect  changes 
in  strategic  business  objectives,  reorganization  or  business  process  changes 

System  development  must  be  robust  to  handle  many  iterations  required  to  refine  processes  and  to  incorporate 
changes  to  the  system 

Development 

Control 

Architecture  baseline  must  be  established  and  organization  must  have  a  clear  understanding  of  the  business 
strategies  that  must  be  supported  by  the  organization's  enterprise  architecture  (Where  we  are) 

Framework  guides  architecture  design  and  investment  decision  making 

Define  the  target  business  view 

Determine  target  architecture  (Where  we  want  to  be) 

Identify  gaps  between  baseline  and  established  targets 

Outline  plan  on  how  to  reach  target  architecture— develop  a  modemization/implementation  plan  (How  we  are 
going  to  get  there) 

Use  an  agreed  upon  system  development  methodology  and  universal  standards 

Start  with  doable  and  critical  system  development  projects 

Operational 

Control 

Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

Business  requirements  drive  the  rest  of  the  target  architectural  views 

Govemace  of  the  architecture  establishes  and  communicates  a  defined  structure  and  clear  roles  and 
responsibilities 

Maintenance 

Control 

Evolve  the  architecture  over  time  in  a  iterative  step  by  step  transition  plan  of  continous  improvement  and  analyze 
how  changes  in  the  organization's  mission,  functions,  and  needs  might  have  an  effect  on  system  development 

Common  understanding  and  conformance  to  architecture  principles  and  standards  leads  to  consitent  enforcement 
of  guidance,  informed  system  development  decisions,  and  reduced  redundancy 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear,  meaningful,  and  quantifiable 

Architecture  changes  must  be  centrally  controlled  and  changes  must  be  directed  through  the  group  responsible  for 
this  task 

Feedback  is  recieved  on  performance  so  future  architecture  changes  will  be  more  successful 

Focus  must  be  maintained  on  business  and  technology  risk  mitigation 

Processes 

Understanding  the  business  processes  allows  the  architecture  to  align  and/or  modify  the  IT  goals  and  objectives 
to  ensure  the  implementation  of  IT  systems  that  will  match  the  required  business  needs 

Organization's  must  make  the  transition  from  function  oriented  management  to  process  based  management 

Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable,  measurable,  and  reduces 
redundancy 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  individual  buinsess  units— best  practice 
processes  can  be  recognized  and  implemented  across  the  entire  organization 

Openness 

Architecture  provides  common  and  centralized  standards  leading  to  an  open  system  where  system  components  can 
be  reused 

An  interoperable  architecture  allows  other  systems  to  easily  integrate  into  it— systems  can  be  moved  in  and  out  of 
the  architecture 

Avoid  intrusive  integration  by  modifying  code  in  legacy  systems— use  data  brokers  to  transform  data  from  one 
format  to  another 

Data 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data  intergration  and  data  sharing  across 
diverse  applications 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the  data  that  is  provided 

Technology 

As  new  computing  models  become  available  an  architecture  is  used  to  make  decisions  in  deciding  to  implement 
new  technologies  and/or  retain  legacy  systems  to  develop  the  best  technological  fit  for  the  organization  so  it  can 
gain  a  competitive  advantage 

A  developed  IT  infrastructure  that  enforces  its  standards  gives  an  organization  the  ability  to  manuever  in  response 
to  market  opportunities 

Organizational  / 
Personnel 

The  enterprise  architecture  must  have  senior  management  support 

Development  of  an  architecture  must  include  the  business/functional  users 

Gain  knowledgeable  architecture  resources  from  consultants 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users  with  the  ability  and 
authority  to  answer  human,  technical,  and  business  questions  and  carry  out  assigned  responsbilities 

A  culture  must  be  developed  that  focuses  on  the  importance  of  coordinated  planning  between  business  and  IT 
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Appendix  C:  Validated  Codebook 


Article  Number 

Key  Issues 

Flexibility 

Architecture  development  must  be  flexible  to  accommodate  a  range  of  architectures  and  functional  areas 
requirements 

Architecture  must  be  capable  of  adapting  or  modifying  itself  to  reflect  changes  in  strategic  objectives, 
reorganization  and/or  business  process  changes 

System  development  must  be  robust  to  handle  the  multiple  iterations  required  to  refine  processes  and  incorporate 
changes  to  the  system 

Development 

Control 

Architecture  baseline  must  be  established  and  the  organization  must  have  a  clear  understanding  of  its  business 
strategies  (Where  we  are) 

Framework  guides  architecture  design  and  investment  decision  making 

Define  the  target  business  view 

Determine  target  architecture  (Where  we  want  to  be) 

Identify  gaps  between  baseline  and  established  targets 

Outline  plan  on  how  to  reach  target  architecture— develop  a  modemization/implementation  plan  (How  are  we 
going  to  get  there) 

Use  an  agreed  upon  system  development  methodology  and  universal  standards 

Start  with  doable  and  critical  system  development  projects 

Operational 

Control 

Architecture  must  be  tied  directly  to  the  organization's  operational  mission  and  vision 

Business  requirements  drive  the  rest  of  the  target  architectural  views 

Governance  of  the  architecture  communicates  a  defined  structure  and  establishes  clear  roles  and  responsibilities 

Maintenance 

Control 

Evolve  the  architecture  over  time  in  a  iterative  step  by  step  transition  plan  and  analyze  how  changes  in  the 
organization's  mission,  functions,  and  needs  might  have  an  effect  on  system  development 

Common  understanding  and  conformance  to  architecture  principles  and  standards  leads  to  consistent  enforcement 
of  guidance,  informed  system  development  decisions,  and  reduced  redundancy 

The  value  added  from  the  architecture  must  be  measured  by  metrics  that  are  clear,  meaningful,  and  quantifiable 

Architecture  changes  must  be  centrally  controlled  and  changes  must  be  directed  through  the  group  responsible  for 
this  task 

Feedback  is  received  on  performance  so  future  architecture  changes  will  be  more  successful 

Focus  must  be  maintained  on  business  and  technology  risk  mitigation 

Processes 

Understanding  the  business  processes  allows  the  architecture  to  ensure  the  implementation  of  IT  systems  that  will 
match  the  required  business  needs 

Organization's  must  make  the  transition  from  function  (stove-piped)  oriented  management  to  process  (horizontal) 
based  management 

Managing  by  processes  allows  architecture  modules  to  become  repeatable,  reusable,  measurable,  and  reduces 
redundancy 

Central  control  of  standardized  processes  allows  for  rapid  innovation  from  individual  business  units— best  practice 
processes  can  be  recognized  and  implemented  across  the  entire  organization 

Openness 

Architecture  provides  common  and  centralized  standards  leading  to  an  open  system  where  system  components  can 
be  reused 

An  interoperable  architecture  allows  other  systems  to  seamlessly  integrate  with  it,  allowing  other  systems  to  be 
moved  in  and  out  of  the  architecture 

Data 

Standardizing  data  definitions  and  data  exchange  procedures  facilitates  data  integration  and  data  sharing  across 
diverse  applications 

Data  owners  must  be  identified  who  are  responsible  for  ensuring  the  integrity  of  the  data  that  is  provided 

Technology 

An  architecture  is  a  tool  that  allows  the  organization  to  gain  a  competitive  by  being  a  tool  that  can  assist  in  making 
the  decision  whether  or  not  to  implement  new  technologies  and/or  retain  legacy  systems 

A  developed  IT  infrastructure  that  enforces  its  standards  gives  an  organization  the  ability  to  maneuver  in  response 
to  market  opportunities 

Organizational  / 
Personnel 

The  enterprise  architecture  must  have  senior  management  support 

Development  of  an  architecture  must  include  the  business/functional  users 

Gain  knowledgeable  architecture  resources  from  consultants 

Select  and  train  a  team  of  enterprise  architects,  governing  bodies  and  functional  users  with  the  ability  and 
authority  to  answer  human,  technical,  and  business  questions  and  carry  out  assigned  responsibilities 

A  culture  must  be  developed  that  focuses  on  the  importance  of  coordinated  planning  between  business  and  IT 
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